Duet 3HC Expansion looses Connection

T3P3Tony

@developeralgo222 What happens if you do the following.

Connect the 6XD to 3HC Board 2, via a known good cable. confirm sync and communications
Unplug the CAN cable at the 3HC end. plug in Board 1, power it up, send M122 B1

developeralgo222

@Phaedrux said in Duet 3HC Expansion looses Connection:

If you leave 3hc1 disconnected are you able to get all the other boards synced with any cable combo?

Yes , Using standard straight Phone cable (4 meters) and phone or self-made CAT 5 & 6 Cables ( 2-pairs used) with RJ11 termination (0.5m)

6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync except 3HC1

but i can't get Kenable ADSL 2+ cables to work in any combination . Checked them and they are OK.

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 What happens if you do the following.

Connect the 6XD to 3HC Board 2, via a known good cable. confirm sync and communications

Unplug the CAN cable at the 3HC end. plug in Board 1, power it up, send M122 B1

CAN Bus connection: 6XD ---> phone cable (4m) --> 3HC1 (CAN Bus Terminated ) === No Sync

2/2/2024, 2:21:45 PM

M122 B1
CAN response timeout: board 1, req type 6024, RID 55


2/2/2024, 2:21:39 PM

M122 B0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.3 (2024-01-24 17:59:29) running on Duet 3 MB6XD v1.01 or later (standalone mode)
Board ID: 0JD2M-999AL-D25S0-7JTD6-3SJ6S-953M2
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 153600
Dynamic ram: 117352 of which 0 recycled
Never used RAM 72184, free system stack 204 words
Tasks: NETWORK(1,ready,14.1%,173) ETHERNET(5,nWait 7,0.1%,467) HEAT(3,nWait 6,0.0%,368) Move(4,nWait 6,0.0%,335) CanReceiv(6,nWait 1,0.0%,797) CanSender(5,nWait 7,0.0%,334) CanClock(7,delaying,0.0%,336) MAIN(1,running,29.1%,444) IDLE(0,ready,56.7%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:01:56 ago, cause: power up
Last software reset at 2024-02-02 14:07, reason: User, Gcodes spinning, available RAM 71260, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 29.1, current 39.9, max 40.4
Supply voltage: min 24.1, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 2024-02-02 14:21:37
Slowest loop: 5.59ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 637, received 333, lost 0, errs 4159263, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 49), ts 582/0/0
Tx timeouts 0,0,581,54,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 9.52ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

it's also not visible on DWC

Screenshot 2024-02-02 142524.png

developeralgo222

This post is deleted!

developeralgo222

@Phaedrux said in Duet 3HC Expansion looses Connection:

If you leave 3hc1 disconnected are you able to get all the other boards synced with any cable combo?

Yes ,

when using a Phone cable ( 4m ) from 6XD to the rest of 3HC boards

6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync

developeralgo222

@developeralgo222

@droftarts @dc42 , I also noticed that the RJ11 Connectors (Sockets ) might be having an issue with its Pins and seem not to have a good contact with the RJ11 male connector. I know you did mention it earlier. It's hard to test but if you use several testing cables in and out of the socket you start to notice that the contact is sometimes no longer working. This might just be a side effect of me doing testing or the BOM might be revisted to have a more reliable part

soare0

@developeralgo222
@soare0

I first deleted my post, as I did not want to offend anyone, even unwillingly, but finally I restored it: maybe it helps though.
Anyway, now it rest one more step, to put the remaining board last, and check if the first 3 (2,3,4) are ok, and the (now) forth (1) is or not syncing.
Kenable ADSL cables are simply pin to pin cables, or so it should. Only if in fact are some crossover cables they should not have worked, you need to check the colors of the wires. Still, it is not impossible to be faulty, somehow. Otherwise, for your setup, (almost) any insulated wire should work.

T3P3Tony

@developeralgo222 ok so it does look like that 3HC needs replacing as well. please follow the same process as before with an email to warranty@duet3d.com. I suspect the 6XD may actually have been ok, but we will have to see when we get ti back for testing.

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 ok so it does look like that 3HC needs replacing as well. please follow the same process as before with an email to warranty@duet3d.com. I suspect the 6XD may actually have been ok, but we will have to see when we get ti back for testing.

The problem is that each type of cable and the Lengths i have tested somehow gives slightly different results on the Test Bench when i test individual 3HC boards with CAN Bus Terminations

6XD =====> 3HC1 or 3HC2 or 3HC3 or 3HC4 each with CAN Bus termination (Individual Tests from main board 6XD to each 3HC only)

Kenable ADSL 2+ Cables ( 15 meters )
3HC1 == No Sync (M122 B1 -- CAN response timeouts) , 3HC2 and 3HC4 == Sync, 3HC3 == Connects with a sync but errors (M122 B3 ) on CAN response timeouts , disconnect & reconnecting every second )
Straight Phone Cables ( 4 meters )
3HC1 = No Sync (M122 B1 -- CAN response timeouts) but 3HC2 , 3HC3 & 3HC4 == Sync with No Errors
Self-Made Cat 5 or 6 with Twisted 2 pairs in use ( 5 meters )
3HC1 == No Sync at all (M122 B1 -- CAN response timeouts) but 3HC2 and 3HC4 == Sync with No Errors and 3HC3 == No Sync at all

I have decided to focus on Kenable ADSL 2+ cables because they have a bit of good insulation to avoid tagging or pulling at the RJ11 termination but Phone Cables (4m) seems to work the best ( at least i can get 3 x 3HC boards working ) .i have tested all the cables and they tested OK.

developeralgo222

@soare0 said in Duet 3HC Expansion looses Connection:

@developeralgo222
My 2c, if allowed, as I followed the topic a little.
I only have a 6XD with 1x3HC, and they work flawlesly, and I am sure 3 of them would, if ok.
I noticed that you are not a hardware engineer, so it is understandable your... technique is not... up to standards, but yet maybe I can help.

For such connections length, termination, shielding, twisting, is almost irrelevant, excepting the case that you would have nearby some very powerful EMI sources.
The worst thing happened was not having common ground, I smiled when noticed, bot also I saw you corrected. It is a must when you work with multiple power supply (not talking about isolation needs, when it is the case, but this is not), as we can easily have a big common mode voltage, wich may be big enough in such cases, either LF, 50Hz, or HF... xxx Hz. So, a big wire between two V- (one of each PS), is the first thing needed to be taking care of, prior to the first power up. After that, connect all the stuff exactly as you did.
Note: voltage differential may cripple some stuff on some boards, if that connexion is not in place, it is simple hazard most of the times.

CAN is reliable enough, do not worry about it. Common mode noise/bias is another thing, and this may be troubling. But not CAN itself, as it compensates for most of this stuff, anyway.

I used simple telephone cable, untwisted, unshielded, and ... no problem. Even crimped myself, with a very doubtful crimping tool. Just take care to crimp hard enough... You will NOT, I repeat, you will NOT have problems with this, there is not special cable for CAN, it is a simple serial connection, with hardware/firmware specific communication protocols.

Just dissasemble all, and redo the test platform, as it were when working. Test each 3HC with 6XD separately, with the same cable. Then whatever one of them, changing cables. Then add the second,, with another (tested) cable, Then add the third, with the third cable. This way you will be able to separate problems.

If all works reliable, put the 6XD in place, and test it. Add just the first board with the first cable, and then build up until the third, retesting at each step.
Take care that you will want to repeatedly change the adresses of the boards, when testing like this, at least when testing each board, to check if each board works well on each address it would eventually be connected.
Build only step by step, even in that ... structure of yours. I very much suspect there is a problem with those metal ..... whatever, or with a cable with a loose connection. I would have the brass risers replaced by some plastic ones, even if, normally it should not matter. Mine are, I think, plated copper/brass, I need to look, maybe in some paranoia excess I changed to PE ones. Again, it should not matter, but again, maybe there is also some groove there, etc...

Never assume it is somebody else fault (bad board, bad configuration, etc), until you can prove it. I am sorry to be blunt, but it is one of the main mistakes you did here, let name it the first. I know the drill, as I observed it SO MANY times in the past. Usually, the mechanic engineer and the electronics engineer, blame each other (most of the times the mechanics blames electronists, and most of the times they are wrong).
So the key to debug this is to check yourself at every step, do it as gradually as possible. I know this is a test engineer behaviour, wich I hardly learn from the best...

The second mistake is that you tried to test all the system at once, without proper tools and knowledge. Now, when it works, it works, hehe, luck is luck...

The third mistake is to assume that you really know what you are doing. I did it many times, too. That wire, man, that wire again...!

Good luck...

Thanks for the tips but i think you are assuming a lot of things. i got a lot of help from this forum. This was my first time using Duet 3 boards on any major project and i wanted to make sure that i did it correctly so that's why i asked a lot of questions and any help. I am not new to electronics / Computer / Hardware engineering. i always welcome any suggestions or help in whatever i do, No offense.

T3P3Tony

@developeralgo222 have you tried the other plug on the 3HCs that are problematic - while one is marked in and the other out they are electrically identical

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 have you tried the other plug on the 3HCs that are problematic - while one is marked in and the other out they are electrically identical

Yes, on each 3HC board i have tested both CAN connection Sockets to confirm( both CAN_IN & CAN_OUT )

Can you confirm what Firmware binary files are required on the 6XD from Github Repo for it work properly. i know that most of this files are not used by 6XD & 3HC unless a board that requires the firmware is connected. Is there a binary file that should not be on the 6XD in this list and would cause issues ?

M20 S2 P"/firmware"

"dir": "/firmware",
    "first": 0,
    "files": [
        "Duet3Firmware_EXP1HCL.bin",
        "Duet3Firmware_EXP1XD.bin",
        "Duet3Firmware_EXP3HC.bin",
        "Duet3Firmware_MB6HC.bin",
        "Duet3Firmware_MB6XD.bin",
        "Duet3Firmware_Mini5plus.uf2",
        "Duet3Firmware_TOOL1LC.bin",
        "Duet3_SDiap32_MB6HC.bin",
        "Duet3_SDiap32_MB6XD.bin",
        "Duet3_SDiap32_Mini5plus.bin",
        "DuetWiFiServer.bin",
        "Duet2CombinedFirmware.bin",
        "Duet2Firmware_SBC.bin",
        "Duet2_SDiap32_Maestro.bin",
        "Duet2_SDiap32_WiFiEth.bin",
        "Duet3Bootloader-SAME5x.bin",
        "Duet3Firmware_M23CL.bin",
        "Duet3Firmware_SAMMYC21.bin",
        "Duet3Firmware_SZP.bin",
        "Duet3Firmware_TOOL1RR.bin",
        "DuetAPI.xml",
        "DuetMaestroFirmware.bin",
        "DuetWiFiModule_32S3.bin",
        "PccbFirmware.bin",
        "Duet3_CANiap32_MB6HC.bin",
        "Duet3_CANiap32_MB6XD.bin",
        "Duet3_CANiap32_Mini5plus.bin"
    ],

T3P3Tony

@developeralgo222

Ok so to focus on 3HC3.:

Poor connections with the "Kenable ADSL 2+ Cables ( 15 meters )"
No connection with the "Self-Made Cat 5 or 6 with Twisted 2 pairs in use ( 5 meters )"
Good connection with the Straight Phone Cables ( 4 meters )
Good connection with the "self-made CAT 5 & 6 Cables ( 2-pairs used) with RJ11 termination (0.5m)"?

And in all cases you have tested multiple cables of the same type with the 3HC3, with nothing else on the bus?

And those same cables work with 3HC2 and 3HC4?

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222

Ok so to focus on 3HC3.:

Poor connections with the "Kenable ADSL 2+ Cables ( 15 meters )"

No connection with the "Self-Made Cat 5 or 6 with Twisted 2 pairs in use ( 5 meters )"

Good connection with the Straight Phone Cables ( 4 meters )

Good connection with the "self-made CAT 5 & 6 Cables ( 2-pairs used) with RJ11 termination (0.5m)"?

And in all cases you have tested multiple cables of the same type with the 3HC3, with nothing else on the bus?

And those same cables work with 3HC2 and 3HC4?

Yes , All cables work with 3HC2 & 3HC4. Phone cable works with 3HC2, 3HC3 & 3HC4 . All cables don't seem to work with 3HC1 at the moment. Going to try and reflash 3HC1 and test it again individually

T3P3Tony

@developeralgo222 yeah i am assuming for all these tests they are done individually for now.

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 yeah i am assuming for all these tests they are done individually for now.

Yes

gloomyandy

@developeralgo222 Just to clarify, when you are using the boards individually are you enabling the termination resistors on the 3HC boards that is being tested?

developeralgo222

@gloomyandy said in Duet 3HC Expansion looses Connection:

@developeralgo222 Just to clarify, when you are using the boards individually are you enabling the termination resistors on the 3HC boards that is being tested?

Yes , Each 3HC board is CAN Bus terminated when testing individually.

developeralgo222

@T3P3Tony , @dc42 , @droftarts

Individual Bench Testing: Interesting discovery

My first 6XD with RRF 3.5 rc3 works with 3HC1 but doesn't work with 3HC2, 3HC3, or 3HC4 but the second replacement 6XD works with 3HC2, 3HC3, or 3HC4 but not 3HC1 . How is that possible. Same cable (Phone Cable (4m) testing)

same RRF 3.5 rc3 firmware and same cable , Same bootloaders Ver 2.8 on 3HC boards. The SD Cards are a copy of each other. They have same files .

Is there anything on 6XD Version 1.01 Board that would cause this kind of issue ?

6XD (Initial Board) ===> 3HC1 (with CAN Bus termination ) ===> Works and Syncs without Errors
6XD (Initial Board) ===> 3HC2 or 3HC3 or 3HC4 (with CAN Bus termination ) ===> Does not Work and Do not Sync
6XD (Replacement Board) ===> 3HC1 (with CAN Bus termination ) ===> Does not Work and Do not Sync
6XD (Replacement Board) ===> 3HC2 or 3HC3 or 3HC4 (with CAN Bus termination ) ===> Works and Syncs without Errors

droftarts

@developeralgo222 Thanks for doing the continued testing. I don't have an explanation for the behaviour, but it seems like there's something amiss with the initial 6XD and 3HC1. Have you already asked for a warranty replacement of 3HC1, as @T3P3Tony suggested in his earlier post here https://forum.duet3d.com/post/333134 ?

Ian