Duet 3HC Expansion looses Connection

developeralgo222

@developeralgo222
@dc42, @droftarts @Phaedrux ,

Received the new 6XD board and started doing the Normal isolated Bench tests that i did earlier TEST1 and TEST 2 (Above) using Kenable ADSL 2+ Cables. Checked the resistors , they are OK

All boards have same Firmware RRF 3.5_rc3 with Expansion boards 3HC running 3.5_rc3 with Version 2.8 Bootloader

Self-made Cables are straight CAT 5 or CAT 6 twisted using 2 pairs with RJ11 termination

Same Test with 0.5m, 4m and 5m Where 6XD--> 0.5m(self-made)/4m(self-made)/5m(straight phone Cable) -->3HC1 -->0.5m(kenable)-->3HC2-->0.5m(kenable)-->3HC3-->0.5m(kenable)-->3HC4
Same Test with 15m Kenable ADSL 2+ cable Where 6XD-->15m(kenable)-->3HC1 -->0.5m(kenable)-->3HC2-->0.5m(kenable)-->3HC3--> 0.5m(kenable) -->3HC4

It does not look good so far, The tests are failing synchronization to the Expansion boards. i need to complete all the tests and then i will post the results.

T3P3Tony

@developeralgo222 are you testing with just 1 £HC on the bus to start with (and then test each 3HC individually)

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 are you testing with just 1 £HC on the bus to start with (and then test each 3HC individually)

Yes,

TEST 1: Completely Isolated Bench Test ---- Only CAN Bus & Power ( VIN & GND ) are connected and CAN Bus Termination Jumpers on the Last Expansion 3HC Board in that Chain

using Kenable ADSL 2+ (1m / 15m) Cable with RJ11 Termination

6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC1 ==== Do Not Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC2 ==== Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC3 ==== Do Not Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC4 ==== Sync

using self-made 0.5m/4m/5m twisted CAT 5 & 6 Cable with RJ11 Termination

6XD ---> self-made 0.5m/4m/5m --->3HC1 ==== Do Not Sync
6XD ---> self-made 0.5m/4m/5m --->3HC2 ==== Sync
6XD ---> self-made 0.5m/4m/5m --->3HC3 ==== Do Not Sync
6XD ---> self-made 0.5m/4m/5m --->3HC4 ==== Sync

using standard straight Phone (4m) cable with RJ11 Termination

6XD ---> 4m (Phone Cable 4m) --->3HC1 ==== Do Not Sync
6XD ---> 4m (Phone Cable 4m) --->3HC2 ==== Sync
6XD ---> 4m (Phone Cable 4m) --->3HC3 ==== Sync
6XD ---> 4m (Phone Cable 4m) --->3HC4 ==== Sync

TEST 2: --being done now

using standard straight Phone cables 4m / 0.5m with RJ11 Termination

6XD ---> 4m ---> 3HC1 ---> 0.5m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Do Not Sync
6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync

2/2/2024, 9:40:03 AM

M122 B2
Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156544, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,175) HEAT(2,nWait 6,0.0%,136) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,64) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,408) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,259), total 100.0%
Owned mutexes:
Last reset 00:02:38 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/4, peak Rx sync delay 182, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.3, current 12.4, max 12.4
MCU temperature: min 34.9C, current 35.6C, max 35.6C
Last sensors broadcast 0x00000000 found 0 113 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 882, send timeouts 0, received 540, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

2/2/2024, 9:39:41 AM

M122 B3
Diagnostics for board 3:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156544, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,175) HEAT(2,nWait 6,0.0%,125) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,64) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,404) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Owned mutexes:
Last reset 00:02:16 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11132, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11131, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11131, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -4/5, peak Rx sync delay 181, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 33.9C, current 34.2C, max 34.5C
Last sensors broadcast 0x00000000 found 0 183 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 726, send timeouts 0, received 443, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0



2/2/2024, 9:39:34 AM

M122 B4
Diagnostics for board 4:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156664, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,180) HEAT(2,nWait 6,0.0%,136) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,65) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,402) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,264), total 100.0%
Owned mutexes:
Last reset 00:02:09 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3937, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3937, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3936, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/5, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.3, current 24.3, max 24.3
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 33.4C, current 33.4C, max 33.8C
Last sensors broadcast 0x00000000 found 0 181 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 710, send timeouts 0, received 434, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

i have flashed 3HC1 bootloader and firmware (just in case its corrupted ) to see if that helps but 3HC1 does not seem to be able to sync.

@dc42 , @droftarts , @T3P3Tony

Is there anything i might be missing on Main 6XD board that might prevent 4 x 3HC boards from syncing ? Since the maximum number of CAN connected boards (in RRF 3.5 ) is 20 for 6XD and that's within the Limit to allow connection of 4 x 3HC boards.

What does Standalone Mode mean in the diagnostic print out M122 B0 . Is there a different mode ?

RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.3 (2024-01-24 17:59:29) running on Duet 3 MB6XD v1.01 or later (standalone mode)

soare0

@developeralgo222
My 2c, if allowed, as I followed the topic a little.
I only have a 6XD with 1x3HC, and they work flawlesly, and I am sure 3 of them would, if ok.
I noticed that you are not a hardware engineer, so it is understandable your... technique is not... up to standards, but yet maybe I can help.

For such connections length, termination, shielding, twisting, is almost irrelevant, excepting the case that you would have nearby some very powerful EMI sources.
The worst thing happened was not having common ground, I smiled when noticed, bot also I saw you corrected. It is a must when you work with multiple power supply (not talking about isolation needs, when it is the case, but this is not), as we can easily have a big common mode voltage, wich may be big enough in such cases, either LF, 50Hz, or HF... xxx Hz. So, a big wire between two V- (one of each PS), is the first thing needed to be taking care of, prior to the first power up. After that, connect all the stuff exactly as you did.
Note: voltage differential may cripple some stuff on some boards, if that connexion is not in place, it is simple hazard most of the times.
CAN is reliable enough, do not worry about it. Common mode noise/bias is another thing, and this may be troubling. But not CAN itself, as it compensates for most of this stuff, anyway.
I used simple telephone cable, untwisted, unshielded, and ... no problem. Even crimped myself, with a very doubtful crimping tool. Just take care to crimp hard enough... You will NOT, I repeat, you will NOT have problems with this, there is not special cable for CAN, it is a simple serial connection, with hardware/firmware specific communication protocols.
Just dissasemble all, and redo the test platform, as it were when working. Test each 3HC with 6XD separately, with the same cable. Then whatever one of them, changing cables. Then add the second,, with another (tested) cable, Then add the third, with the third cable. This way you will be able to separate problems.
If all works reliable, put the 6XD in place, and test it. Add just the first board with the first cable, and then build up until the third, retesting at each step.
Take care that you will want to repeatedly change the adresses of the boards, when testing like this, at least when testing each board, to check if each board works well on each address it would eventually be connected.
Build only step by step, even in that ... structure of yours. I very much suspect there is a problem with those metal ..... whatever, or with a cable with a loose connection. I would have the brass risers replaced by some plastic ones, even if, normally it should not matter. Mine are, I think, plated copper/brass, I need to look, maybe in some paranoia excess I changed to PE ones. Again, it should not matter, but again, maybe there is also some groove there, etc...
Never assume it is somebody else fault (bad board, bad configuration, etc), until you can prove it. I am sorry to be blunt, but it is one of the main mistakes you did here, let name it the first. I know the drill, as I observed it SO MANY times in the past. Usually, the mechanic engineer and the electronics engineer, blame each other (most of the times the mechanics blames electronists, and most of the times they are wrong).
So the key to debug this is to check yourself at every step, do it as gradually as possible. I know this is a test engineer behaviour, wich I hardly learn from the best...
The second mistake is that you tried to test all the system at once, without proper tools and knowledge. Now, when it works, it works, hehe, luck is luck...
The third mistake is to assume that you really know what you are doing. I did it many times, too. That wire, man, that wire again...!

Good luck...

Phaedrux

@developeralgo222 said in Duet 3HC Expansion looses Connection:

What does Standalone Mode mean in the diagnostic print out M122 B0 . Is there a different mode ?

Standalone means without a raspberry Pi.

T3P3Tony

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Is there anything i might be missing on Main 6XD board that might prevent 4 x 3HC boards from syncing

No, 4x3HCs should work.

Looks like it was B1 that was causing the issues

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Is there anything i might be missing on Main 6XD board that might prevent 4 x 3HC boards from syncing

No, 4x3HCs should work.

Looks like it was B1 that was causing the issues.

Is there an easier way to identify if the issue is the 3HC1 Board itself or the CAN Bus connectors on 3HC1 board. If the issue is the 3HC Board itself then i would like to find out and have it swapped.

Has anyone in the Duet Forum actually tested something similar to what i have configured

6XD MainBoard with 4 x 3HC boards

Please any pointers that i have not done yet ?

I am able to get it to Sync with 1 , 2 or 3 Duet 3HC boards (Even though not very reliable ) but i can't get it work with 4 x 3HC boards. I don't believe that cables are the issue. Kenable ADSL 2+ Cables which are recommended do not work even on the test Bench setup that seems to sync when using a phone cable (4m) .

6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync

Phaedrux

If you leave 3hc1 disconnected are you able to get all the other boards synced with any cable combo?

T3P3Tony

@developeralgo222 What happens if you do the following.

Connect the 6XD to 3HC Board 2, via a known good cable. confirm sync and communications
Unplug the CAN cable at the 3HC end. plug in Board 1, power it up, send M122 B1

developeralgo222

@Phaedrux said in Duet 3HC Expansion looses Connection:

If you leave 3hc1 disconnected are you able to get all the other boards synced with any cable combo?

Yes , Using standard straight Phone cable (4 meters) and phone or self-made CAT 5 & 6 Cables ( 2-pairs used) with RJ11 termination (0.5m)

6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync except 3HC1

but i can't get Kenable ADSL 2+ cables to work in any combination . Checked them and they are OK.

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 What happens if you do the following.

Connect the 6XD to 3HC Board 2, via a known good cable. confirm sync and communications

Unplug the CAN cable at the 3HC end. plug in Board 1, power it up, send M122 B1

CAN Bus connection: 6XD ---> phone cable (4m) --> 3HC1 (CAN Bus Terminated ) === No Sync

2/2/2024, 2:21:45 PM

M122 B1
CAN response timeout: board 1, req type 6024, RID 55


2/2/2024, 2:21:39 PM

M122 B0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.3 (2024-01-24 17:59:29) running on Duet 3 MB6XD v1.01 or later (standalone mode)
Board ID: 0JD2M-999AL-D25S0-7JTD6-3SJ6S-953M2
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 153600
Dynamic ram: 117352 of which 0 recycled
Never used RAM 72184, free system stack 204 words
Tasks: NETWORK(1,ready,14.1%,173) ETHERNET(5,nWait 7,0.1%,467) HEAT(3,nWait 6,0.0%,368) Move(4,nWait 6,0.0%,335) CanReceiv(6,nWait 1,0.0%,797) CanSender(5,nWait 7,0.0%,334) CanClock(7,delaying,0.0%,336) MAIN(1,running,29.1%,444) IDLE(0,ready,56.7%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:01:56 ago, cause: power up
Last software reset at 2024-02-02 14:07, reason: User, Gcodes spinning, available RAM 71260, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 29.1, current 39.9, max 40.4
Supply voltage: min 24.1, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 2024-02-02 14:21:37
Slowest loop: 5.59ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 637, received 333, lost 0, errs 4159263, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 49), ts 582/0/0
Tx timeouts 0,0,581,54,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 9.52ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

it's also not visible on DWC

Screenshot 2024-02-02 142524.png

developeralgo222

This post is deleted!

developeralgo222

@Phaedrux said in Duet 3HC Expansion looses Connection:

If you leave 3hc1 disconnected are you able to get all the other boards synced with any cable combo?

Yes ,

when using a Phone cable ( 4m ) from 6XD to the rest of 3HC boards

6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync

developeralgo222

@developeralgo222

@droftarts @dc42 , I also noticed that the RJ11 Connectors (Sockets ) might be having an issue with its Pins and seem not to have a good contact with the RJ11 male connector. I know you did mention it earlier. It's hard to test but if you use several testing cables in and out of the socket you start to notice that the contact is sometimes no longer working. This might just be a side effect of me doing testing or the BOM might be revisted to have a more reliable part

soare0

@developeralgo222
@soare0

I first deleted my post, as I did not want to offend anyone, even unwillingly, but finally I restored it: maybe it helps though.
Anyway, now it rest one more step, to put the remaining board last, and check if the first 3 (2,3,4) are ok, and the (now) forth (1) is or not syncing.
Kenable ADSL cables are simply pin to pin cables, or so it should. Only if in fact are some crossover cables they should not have worked, you need to check the colors of the wires. Still, it is not impossible to be faulty, somehow. Otherwise, for your setup, (almost) any insulated wire should work.

T3P3Tony

@developeralgo222 ok so it does look like that 3HC needs replacing as well. please follow the same process as before with an email to warranty@duet3d.com. I suspect the 6XD may actually have been ok, but we will have to see when we get ti back for testing.

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 ok so it does look like that 3HC needs replacing as well. please follow the same process as before with an email to warranty@duet3d.com. I suspect the 6XD may actually have been ok, but we will have to see when we get ti back for testing.

The problem is that each type of cable and the Lengths i have tested somehow gives slightly different results on the Test Bench when i test individual 3HC boards with CAN Bus Terminations

6XD =====> 3HC1 or 3HC2 or 3HC3 or 3HC4 each with CAN Bus termination (Individual Tests from main board 6XD to each 3HC only)

Kenable ADSL 2+ Cables ( 15 meters )
3HC1 == No Sync (M122 B1 -- CAN response timeouts) , 3HC2 and 3HC4 == Sync, 3HC3 == Connects with a sync but errors (M122 B3 ) on CAN response timeouts , disconnect & reconnecting every second )
Straight Phone Cables ( 4 meters )
3HC1 = No Sync (M122 B1 -- CAN response timeouts) but 3HC2 , 3HC3 & 3HC4 == Sync with No Errors
Self-Made Cat 5 or 6 with Twisted 2 pairs in use ( 5 meters )
3HC1 == No Sync at all (M122 B1 -- CAN response timeouts) but 3HC2 and 3HC4 == Sync with No Errors and 3HC3 == No Sync at all

I have decided to focus on Kenable ADSL 2+ cables because they have a bit of good insulation to avoid tagging or pulling at the RJ11 termination but Phone Cables (4m) seems to work the best ( at least i can get 3 x 3HC boards working ) .i have tested all the cables and they tested OK.

developeralgo222

@soare0 said in Duet 3HC Expansion looses Connection:

@developeralgo222
My 2c, if allowed, as I followed the topic a little.
I only have a 6XD with 1x3HC, and they work flawlesly, and I am sure 3 of them would, if ok.
I noticed that you are not a hardware engineer, so it is understandable your... technique is not... up to standards, but yet maybe I can help.

For such connections length, termination, shielding, twisting, is almost irrelevant, excepting the case that you would have nearby some very powerful EMI sources.
The worst thing happened was not having common ground, I smiled when noticed, bot also I saw you corrected. It is a must when you work with multiple power supply (not talking about isolation needs, when it is the case, but this is not), as we can easily have a big common mode voltage, wich may be big enough in such cases, either LF, 50Hz, or HF... xxx Hz. So, a big wire between two V- (one of each PS), is the first thing needed to be taking care of, prior to the first power up. After that, connect all the stuff exactly as you did.
Note: voltage differential may cripple some stuff on some boards, if that connexion is not in place, it is simple hazard most of the times.

CAN is reliable enough, do not worry about it. Common mode noise/bias is another thing, and this may be troubling. But not CAN itself, as it compensates for most of this stuff, anyway.

I used simple telephone cable, untwisted, unshielded, and ... no problem. Even crimped myself, with a very doubtful crimping tool. Just take care to crimp hard enough... You will NOT, I repeat, you will NOT have problems with this, there is not special cable for CAN, it is a simple serial connection, with hardware/firmware specific communication protocols.

Just dissasemble all, and redo the test platform, as it were when working. Test each 3HC with 6XD separately, with the same cable. Then whatever one of them, changing cables. Then add the second,, with another (tested) cable, Then add the third, with the third cable. This way you will be able to separate problems.

If all works reliable, put the 6XD in place, and test it. Add just the first board with the first cable, and then build up until the third, retesting at each step.
Take care that you will want to repeatedly change the adresses of the boards, when testing like this, at least when testing each board, to check if each board works well on each address it would eventually be connected.
Build only step by step, even in that ... structure of yours. I very much suspect there is a problem with those metal ..... whatever, or with a cable with a loose connection. I would have the brass risers replaced by some plastic ones, even if, normally it should not matter. Mine are, I think, plated copper/brass, I need to look, maybe in some paranoia excess I changed to PE ones. Again, it should not matter, but again, maybe there is also some groove there, etc...

Never assume it is somebody else fault (bad board, bad configuration, etc), until you can prove it. I am sorry to be blunt, but it is one of the main mistakes you did here, let name it the first. I know the drill, as I observed it SO MANY times in the past. Usually, the mechanic engineer and the electronics engineer, blame each other (most of the times the mechanics blames electronists, and most of the times they are wrong).
So the key to debug this is to check yourself at every step, do it as gradually as possible. I know this is a test engineer behaviour, wich I hardly learn from the best...

The second mistake is that you tried to test all the system at once, without proper tools and knowledge. Now, when it works, it works, hehe, luck is luck...

The third mistake is to assume that you really know what you are doing. I did it many times, too. That wire, man, that wire again...!

Good luck...

Thanks for the tips but i think you are assuming a lot of things. i got a lot of help from this forum. This was my first time using Duet 3 boards on any major project and i wanted to make sure that i did it correctly so that's why i asked a lot of questions and any help. I am not new to electronics / Computer / Hardware engineering. i always welcome any suggestions or help in whatever i do, No offense.

T3P3Tony

@developeralgo222 have you tried the other plug on the 3HCs that are problematic - while one is marked in and the other out they are electrically identical

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 have you tried the other plug on the 3HCs that are problematic - while one is marked in and the other out they are electrically identical

Yes, on each 3HC board i have tested both CAN connection Sockets to confirm( both CAN_IN & CAN_OUT )

Can you confirm what Firmware binary files are required on the 6XD from Github Repo for it work properly. i know that most of this files are not used by 6XD & 3HC unless a board that requires the firmware is connected. Is there a binary file that should not be on the 6XD in this list and would cause issues ?

M20 S2 P"/firmware"

"dir": "/firmware",
    "first": 0,
    "files": [
        "Duet3Firmware_EXP1HCL.bin",
        "Duet3Firmware_EXP1XD.bin",
        "Duet3Firmware_EXP3HC.bin",
        "Duet3Firmware_MB6HC.bin",
        "Duet3Firmware_MB6XD.bin",
        "Duet3Firmware_Mini5plus.uf2",
        "Duet3Firmware_TOOL1LC.bin",
        "Duet3_SDiap32_MB6HC.bin",
        "Duet3_SDiap32_MB6XD.bin",
        "Duet3_SDiap32_Mini5plus.bin",
        "DuetWiFiServer.bin",
        "Duet2CombinedFirmware.bin",
        "Duet2Firmware_SBC.bin",
        "Duet2_SDiap32_Maestro.bin",
        "Duet2_SDiap32_WiFiEth.bin",
        "Duet3Bootloader-SAME5x.bin",
        "Duet3Firmware_M23CL.bin",
        "Duet3Firmware_SAMMYC21.bin",
        "Duet3Firmware_SZP.bin",
        "Duet3Firmware_TOOL1RR.bin",
        "DuetAPI.xml",
        "DuetMaestroFirmware.bin",
        "DuetWiFiModule_32S3.bin",
        "PccbFirmware.bin",
        "Duet3_CANiap32_MB6HC.bin",
        "Duet3_CANiap32_MB6XD.bin",
        "Duet3_CANiap32_Mini5plus.bin"
    ],