Duet 3HC Expansion looses Connection

[developeralgo222]

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 Yes, that is correct. The three V- terminals on each power supply are basically one output, they are linked to the same rail. Use a decent sized gauge of wire, at least as big as the other power wires.

I asked @dc42 the other day if your problem might be down to an issue with shared ground. I hadn't spotted that you are using two PSUs. However, I thought the CAN bus was pretty tolerant of this sort of thing, but maybe not.

Ian

connected the cable ,now they are syncing without any timeouts

M122  B1
Diagnostics for board 1:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,138) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,413) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:00:42 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 35139, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 35139, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 35140, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -4/5, peak Rx sync delay 156, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 17.9C, current 20.5C, max 20.5C
Last sensors broadcast 0x00000000 found 0 142 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 364, send timeouts 0, received 221, lost 0, errs 54, boc 0, free buffers 38, min 38, error reg 360000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
ok

M122  B2

Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,436) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:00:46 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 48746, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 48746, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 48747, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -5/7, peak Rx sync delay 157, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.3, current 24.3, max 24.3
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 21.1C, current 23.9C, max 23.9C
Last sensors broadcast 0x00000000 found 0 133 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 392, send timeouts 0, received 239, lost 0, errs 57, boc 0, free buffers 38, min 38, error reg 3a0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
ok

M122  B3

Diagnostics for board 3:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,424) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:00:53 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 10560, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 10560, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 10561, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -7/6, peak Rx sync delay 155, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 21.1C, current 23.8C, max 23.8C
Last sensors broadcast 0x00000000 found 0 143 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 448, send timeouts 0, received 275, lost 0, errs 60, boc 0, free buffers 38, min 38, error reg 3c0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
ok

M122  B4

Diagnostics for board 4:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 204 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,429) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:00:55 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21146, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21147, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21147, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -5/4, peak Rx sync delay 153, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.3, max 24.3
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 21.6C, current 24.4C, max 24.4C
Last sensors broadcast 0x00000000 found 0 110 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 470, send timeouts 0, received 287, lost 0, errs 63, boc 0, free buffers 38, min 38, error reg 3f0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
ok

M122  B0

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode)
Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2
Used output buffers: 1 of 40 (7 max)
=== RTOS ===
Static ram: 153284
Dynamic ram: 117344 of which 0 recycled
Never used RAM 72508, free system stack 176 words
Tasks: NETWORK(1,ready,105.3%,182) ETHERNET(5,nWait,0.0%,312) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,342) MAIN(1,running,226.8%,444) IDLE(0,ready,2.7%,30), total 334.8%
Owned mutexes: USB(MAIN)
=== Platform ===
Last reset 02:16:05 ago, cause: power up
Last software reset at 2024-01-15 20:32, reason: User, Gcodes spinning, available RAM 72304, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 25.5, current 42.1, max 42.7
Supply voltage: min 0.1, current 24.0, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 0.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 1970-01-01 00:00:00
Slowest loop: 21.40ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.5ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is ready with "M122  B0" in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 40919, received 2240, lost 0, errs 39255999, boc 1404
Longest wait 2ms for reply type 6024, peak Tx sync delay 22911, free buffers 50 (min 49), ts 40828/341/0
Tx timeouts 0,0,39777,47,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 0.47ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 0 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 1 0 0
Socket states: 2 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0
ok

[droftarts]

@developeralgo222 Connecting the grounds of different PSUs is already mentioned in the 'CAN connection basics' here: https://docs.duet3d.com/en/User_manual/Machine_configuration/CAN_connection#grounding

As it would appear this has been the main issue for you, do you have any suggestions to make this section more visible?

Ian

[developeralgo222]

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 Connecting the grounds of different PSUs is already mentioned in the 'CAN connection basics' here: https://docs.duet3d.com/en/User_manual/Machine_configuration/CAN_connection#grounding

As it would appear this has been the main issue for you, do you have any suggestions to make this section more visible?

Ian

That's ok but its a little generic , Feel free to use the Sample illustration that is in this thread to make it clear for users. currently Most DC power supplies used in 3D Printing tend to have upto 3 x pair of DC outputs( V+ & V- ) so making it clear and a critical check for installation would have saved a lot of time. My suggestion: i would add the following below it

Example 1: Powering Duets using a Single DC Power supply

• For Single DC power supply with 3 pairs of DC outputs ( 3 x V+ & 3 x V- )
  • you can connect upto 3 x Duet boards ( i.e 1 Board per terminal pair )

Example 2: Powering Duets using Two DC Power supplies

• For Two DC power supplies with 3 pairs of DC outputs each
  • Required: Connect any one of the V- (Negative) DC output Terminal from PSU 1 to any one of the V- (Negative) DC output Terminal on PSU 2. Also use appropriate wire with slightly larger gauge e.g 14, 16, or 18 AWG
  • You can then connect upto 6 Duet Boards ( i.e 1 Board per terminal pair)

[infiniteloop]

@developeralgo222

you can connect upto 3 x Duet boards ( i.e 1 Board per terminal pair )

Aren’t your examples a bit over-specific?

1. You can’t determine the max. number of attached Duets by counting the outputs of a PSU. Instead, this is a function of the PSU’s power and the power requirements of the Duet and attached hardware (heaters, steppers, etc.).

2. For the same reasons, you can’t tell the people they’d need to add a second PSU as soon as the outputs of the first PSU are occupied. This, too, depends solely on the specs of the PSU(s).

3. If you have to supply multiple voltages to your printer (I have a separate PSU for the 5V rail of my Duet), the V– outputs of the PSUs have to be connected as well. Better yet: all PSU outputs must share a common GND (not EARTH!). The usage of the term GND instead of V– becomes obvious if you want to add a negative voltage to your zoo - in this case, V+ of the PSU for the negative voltage is connected to common GND.

4. Beware of AWG recommendations. Picking 18 AWG from your 2nd example, I would have burnt my printer to ashes. It all depends on the length of the lines and the max. ampere draw of the attached components.

Having said this, I fully agree with your proposal to include sample illustrations or schematics to clarify things. However, @droftarts faces a dilemma: there is no simple pic to illustrate all combinations of DC power outlets and how to wire them - two identical PSUs, two or more different voltages, possibly including a negative one… OK, multiple pics. But what if a user then picks the wrong scheme because the red and blue lines look somehow familiar to him?

The Duet documentation is already quite extensive and detailed, its sheer volume would overwhelm you if it had to cover everything we assume to be "basic knowledge" in the realms of electricity, mechanics, electronics and physics. Instead, the Duet people provide this forum - which is really smart as it offers you the assistance of numerous engineers, scientists, experts and specialists for free.

[developeralgo222]

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

you can connect upto 3 x Duet boards ( i.e 1 Board per terminal pair )

Aren’t your examples a bit over-specific?

1. You can’t determine the max. number of attached Duets by counting the outputs of a PSU. Instead, this is a function of the PSU’s power and the power requirements of the Duet and attached hardware (heaters, steppers, etc.).

2. For the same reasons, you can’t tell the people they’d need to add a second PSU as soon as the outputs of the first PSU are occupied. This, too, depends solely on the specs of the PSU(s).

3. If you have to supply multiple voltages to your printer (I have a separate PSU for the 5V rail of my Duet), the V– outputs of the PSUs have to be connected as well. Better yet: all PSU outputs must share a common GND (not EARTH!). The usage of the term GND instead of V– becomes obvious if you want to add a negative voltage to your zoo - in this case, V+ of the PSU for the negative voltage is connected to common GND.

4. Beware of AWG recommendations. Picking 18 AWG from your 2nd example, I would have burnt my printer to ashes. It all depends on the length of the lines and the max. ampere draw of the attached components.

Having said this, I fully agree with your proposal to include sample illustrations or schematics to clarify things. However, @droftarts faces a dilemma: there is no simple pic to illustrate all combinations of DC power outlets and how to wire them - two identical PSUs, two or more different voltages, possibly including a negative one… OK, multiple pics. But what if a user then picks the wrong scheme because the red and blue lines look somehow familiar to him?

The Duet documentation is already quite extensive and detailed, its sheer volume would overwhelm you if it had to cover everything we assume to be "basic knowledge" in the realms of electricity, mechanics, electronics and physics. Instead, the Duet people provide this forum - which is really smart as it offers you the assistance of numerous engineers, scientists, experts and specialists for free.

i think just a good example is fine

[developeralgo222]

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

you can connect upto 3 x Duet boards ( i.e 1 Board per terminal pair )

Aren’t your examples a bit over-specific?

1. You can’t determine the max. number of attached Duets by counting the outputs of a PSU. Instead, this is a function of the PSU’s power and the power requirements of the Duet and attached hardware (heaters, steppers, etc.).

2. For the same reasons, you can’t tell the people they’d need to add a second PSU as soon as the outputs of the first PSU are occupied. This, too, depends solely on the specs of the PSU(s).

3. If you have to supply multiple voltages to your printer (I have a separate PSU for the 5V rail of my Duet), the V– outputs of the PSUs have to be connected as well. Better yet: all PSU outputs must share a common GND (not EARTH!). The usage of the term GND instead of V– becomes obvious if you want to add a negative voltage to your zoo - in this case, V+ of the PSU for the negative voltage is connected to common GND.

4. Beware of AWG recommendations. Picking 18 AWG from your 2nd example, I would have burnt my printer to ashes. It all depends on the length of the lines and the max. ampere draw of the attached components.

Having said this, I fully agree with your proposal to include sample illustrations or schematics to clarify things. However, @droftarts faces a dilemma: there is no simple pic to illustrate all combinations of DC power outlets and how to wire them - two identical PSUs, two or more different voltages, possibly including a negative one… OK, multiple pics. But what if a user then picks the wrong scheme because the red and blue lines look somehow familiar to him?

The Duet documentation is already quite extensive and detailed, its sheer volume would overwhelm you if it had to cover everything we assume to be "basic knowledge" in the realms of electricity, mechanics, electronics and physics. Instead, the Duet people provide this forum - which is really smart as it offers you the assistance of numerous engineers, scientists, experts and specialists for free.

i think just a good example is fine.

[developeralgo222]

@developeralgo222
@dc42 , @droftarts
Noticing an interesting issue, when i moved the test setup 6XD plus 4x 3HC to the PNP Machine with only Power and CAN Bus Cables connected as before , it doesn't sync at all . The only thing that i changed is the length of the cable from 6XD to 3HC1( from 0.5 ft (shielded ) to 10 ft(shielded) . Everything else is the same , Nothing is plugged onto the boards except PWR & CAN Bus cables.

Hmmmm!!! Not sure what to make of it now. I am thinking EMI issue. Most of my power & Comm cables are shielded cables.

Does CAN Bus has a cable length limitation ?

[droftarts]

@developeralgo222 The cables you have made up don't appear to fit into the RJ11 strain relief, so I'm not sure how much abuse (and I guess you've wiggled them around a lot recently) they are going to take before you start getting poor connections. As at least some of the problem has been solved by connecting V- of the PSUs, maybe try the pre-built cables you originally used, I think successfully?

Max cable length depends on the data rate and the number of devices on the bus. We use 1Mbits/sec as the base data rate. Page 5 of http://www.ti.com/lit/an/slla270/slla270.pdf suggests 1Mbits/sec should be OK to 40m length. The data rate is configurable, see https://docs.duet3d.com/en/User_manual/Reference/Gcodes#m952-set-can-fd-expansion-board-address-andor-normal-data-rate. We have users with very large machines, some with CAN bus longer than 10m, without issue.

As for EMI, I think the whole point of CAN is that it is resilient to noise. Additionally, cable shielding only works if one end of the shielding is grounded. Grounding both ends can cause signals in the shielding itself, and grounding neither end can cause reflections from the cable itself. The 6XD has a shield ground point by the Ethernet socket, but not by the CAN socket. I'm not sure if you can use any GND pin on the board for shield grounding, or if shielding has any special requirements.

Ian

[developeralgo222]

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 The cables you have made up don't appear to fit into the RJ11 strain relief, so I'm not sure how much abuse (and I guess you've wiggled them around a lot recently) they are going to take before you start getting poor connections. As at least some of the problem has been solved by connecting V- of the PSUs, maybe try the pre-built cables you originally used, I think successfully?

Max cable length depends on the data rate and the number of devices on the bus. We use 1Mbits/sec as the base data rate. Page 5 of http://www.ti.com/lit/an/slla270/slla270.pdf suggests 1Mbits/sec should be OK to 40m length. The data rate is configurable, see https://docs.duet3d.com/en/User_manual/Reference/Gcodes#m952-set-can-fd-expansion-board-address-andor-normal-data-rate. We have users with very large machines, some with CAN bus longer than 10m, without issue.

As for EMI, I think the whole point of CAN is that it is resilient to noise. Additionally, cable shielding only works if one end of the shielding is grounded. Grounding both ends can cause signals in the shielding itself, and grounding neither end can cause reflections from the cable itself. The 6XD has a shield ground point by the Ethernet socket, but not by the CAN socket. I'm not sure if you can use any GND pin on the board for shield grounding, or if shielding has any special requirements.

Ian

As for the CAN Bus cables i have checked and double checked them. I do have Fluke professional Cable tester that i use and they are ok. No issues there.

I have no idea what the issue might be now. I have done every possible test that is recommended, rechecked every connection again and no working setup. I don't believe All 4 x Duet 3HC can be defective ? The only thing that comes to mind is that the Duet CAN Bus might be unreliable and not as tolerant as its expected to be

[infiniteloop]

@developeralgo222 To summarise, you got all Duets to sync on your test setup? They failed to sync completely when mounted on the PnP machine? You assume the only difference between both setups to be the CAN wiring - which you verified with your Fluke?

At this point, it is safe to state that all Duets (the 6XD and 4x 3HC) work as expected - that’s proven by your test setup.

The only thing that comes to mind is that the Duet CAN Bus might be unreliable and not as tolerant as its expected to be

That’s exactly the single one topic to be ruled out for sure: with its dedicated controllers and a robust protocol, CAN is one of the most reliable communication means, even in harsh environments.

The CAN cables on the PnP machine are tested good, but some issues still may exist: intermitted contact of the connectors, CAN bus termination, topology of your CAN wiring. To properly commission your setup, add one 3HC at a time and verify the sync. If it doesn’t work, replace the cable (temporarily) and check proper termination.

As was the case with the initial fault - missing GND connection between the two PSUs - there might be „hidden“ issues lurking around as well. These are difficult to diagnose from a distance, so we have to rely on you to tell us the potentially relevant facts. One such example that comes to my mind is an improper mix of GND and Earth: this can lead to quite ”interesting” effects but is extremely difficult to spot.

One question in this context: how do you provide VIN and GND to the 3HCs? With two lines each: PSU -> 3HC, with a common V– line, or V– on the machine’s frame?

But first things first: Commissioning! Please add one 3HC at a time and report your findings.

Good luck,
mb

[developeralgo222]

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222 To summarise, you got all Duets to sync on your test setup? They failed to sync completely when mounted on the PnP machine? You assume the only difference between both setups to be the CAN wiring - which you verified with your Fluke?

At this point, it is safe to state that all Duets (the 6XD and 4x 3HC) work as expected - that’s proven by your test setup.

The only thing that comes to mind is that the Duet CAN Bus might be unreliable and not as tolerant as its expected to be

That’s exactly the single one topic to be ruled out for sure: with its dedicated controllers and a robust protocol, CAN is one of the most reliable communication means, even in harsh environments.

The CAN cables on the PnP machine are tested good, but some issues still may exist: intermitted contact of the connectors, CAN bus termination, topology of your CAN wiring. To properly commission your setup, add one 3HC at a time and verify the sync. If it doesn’t work, replace the cable (temporarily) and check proper termination.

As was the case with the initial fault - missing GND connection between the two PSUs - there might be „hidden“ issues lurking around as well. These are difficult to diagnose from a distance, so we have to rely on you to tell us the potentially relevant facts. One such example that comes to my mind is an improper mix of GND and Earth: this can lead to quite ”interesting” effects but is extremely difficult to spot.

One question in this context: how do you provide VIN and GND to the 3HCs? With two lines each: PSU -> 3HC, with a common V– line, or V– on the machine’s frame?

But first things first: Commissioning! Please add one 3HC at a time and report your findings.

Good luck,
mb

I know that CAN BUs protocol itself is very reliable from my expeience. I tested and added one board at a time as per previous posts. I have 2 x PSUs (24V,20A ) connected as per suggestions above & diagram above. Each of the VIN & GND from each 3HC to goes individual terminals pair on the PSUs ( i have 6XD, 3HC1, & 3HC2 on PSU1 and 3HC3 , 3HC4 on PSU2 )

After the test setup , moved everything to PnP Machine. As indicated above i tested each individual expansion 3HC board( 4 boards in total ) with nothing connected to them only power.

6XD --> Board 1 (with CAN Jumpers) CAN_IN / CAN_OUT -- Sync

6XD --> Board 2 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

6XD --> Board 3 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

6XD --> Board 4 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

FYI:

NOTE: i know it shouldn't be connected this way and it should not work . The CAN Bus Termination jumpers should only be on the Last 3HC board (which is board 4 in my case) in the Chain but to my surprise when i leave the CAN Bus termination Jumpers on each board and connect them , they sync without any issues

@dc42, @infiniteloop, @droftarts , What's happening here , is this even supposed to work ? Termination on each board in the CAN Bus chain works ?

1/19/2024, 9:51:42 AM:

M122 B0: 
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode)
Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 153284
Dynamic ram: 117416 of which 0 recycled
Never used RAM 72436, free system stack 191 words
Tasks: NETWORK(1,ready,167.6%,182) ETHERNET(5,nWait,0.0%,317) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,343) MAIN(1,running,349.0%,444) IDLE(0,ready,4.1%,30), total 520.7%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 01:58:07 ago, cause: power up
Last software reset at 2024-01-18 17:43, reason: User, Gcodes spinning, available RAM 72856, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 25.6, current 41.5, max 42.8
Supply voltage: min 0.3, current 24.0, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 0.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 8 queued, 8 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 2024-01-19 09:51:42
Slowest loop: 10.11ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.4ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 35524, received 8481, lost 0, errs 64909191, boc 60488
Longest wait 0ms for reply type 0, peak Tx sync delay 17202, free buffers 50 (min 49), ts 35439/1747/0
Tx timeouts 0,0,3438,0,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 5.67ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 1 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0


1/19/2024, 9:51:45 AM: 

M122 B1: 
Diagnostics for board 1:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.1%,399) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:09:46 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60274, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60275, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60275, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -5/7, peak Rx sync delay 180, resyncs 1/0, no timer interrupt scheduled
VIN voltage: min 24.1, current 24.1, max 24.2
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 19.4C, current 28.7C, max 28.7C
Last sensors broadcast 0x00000000 found 0 149 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 3092, send timeouts 232, received 1773, lost 0, errs 706207, boc 134, free buffers 38, min 38, error reg ff0000
Last cancelled message type 4511 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0



1/19/2024, 9:51:47 AM: 

M122 B2: 
Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,114) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.1%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:09:49 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 2629, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 2630, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 2631, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -7/4, peak Rx sync delay 182, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.3
V12 voltage: min 12.3, current 12.4, max 12.4
MCU temperature: min 22.1C, current 32.3C, max 32.3C
Last sensors broadcast 0x00000000 found 0 244 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 2486, send timeouts 319, received 1335, lost 0, errs 2290420, boc 2, free buffers 38, min 38, error reg ff0000
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/19/2024, 9:51:50 AM: 

M122 B3: 
Diagnostics for board 3:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,114) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.1%,421) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:09:51 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 12506, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 12507, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 12508, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -4/5, peak Rx sync delay 178, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 21.9C, current 31.1C, max 31.3C
Last sensors broadcast 0x00000000 found 0 171 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 2616, send timeouts 303, received 1263, lost 0, errs 1911382, boc 244, free buffers 38, min 38, error reg ff0000
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/19/2024, 9:51:54 AM: 

M122 B4: 
Diagnostics for board 4:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,114) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.0%,399) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:09:55 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 26004, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 26005, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 26006, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -7/5, peak Rx sync delay 177, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.3
V12 voltage: min 10.9, current 12.4, max 12.4
MCU temperature: min 21.9C, current 31.5C, max 31.5C
Last sensors broadcast 0x00000000 found 0 235 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 3284, send timeouts 213, received 1169, lost 0, errs 5792377, boc 1162, free buffers 38, min 38, error reg ff0000
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

As per CAN Bus Protocol documentation, The CAN bus must be properly terminated using a 120 Ohms CAN bus termination resistor at each end of the bus.
Looking at the explanations on CAN Bus, The resistance of the CAN bus terminal is usually 120 ohms. Two 60 ohm resistors are linked in series in the design, and there are usually two 120 ohm nodes on the CAN bus. This is how i think of it

[dc42]

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Noticing an interesting issue, when i moved the test setup 6XD plus 4x 3HC to the PNP Machine with only Power and CAN Bus Cables connected as before , it doesn't sync at all . The only thing that i changed is the length of the cable from 6XD to 3HC1( from 0.5 ft (shielded ) to 10 ft(shielded) .

Almost certainly the 10ft cable is faulty. Have you checked that it doesn't have crossed connections between the two ends?

@developeralgo222 said in Duet 3HC Expansion looses Connection:

After the test setup , moved everything to PnP Machine. As indicated above i tested each individual expansion 3HC board( 4 boards in total ) with nothing connected to them only power.
6XD --> Board 1 (with CAN Jumpers) CAN_IN / CAN_OUT -- Sync
6XD --> Board 2 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync
6XD --> Board 3 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync
6XD --> Board 4 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

Did you use the same cable between the 6XD and the 3HC in all cases?

[developeralgo222]

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Noticing an interesting issue, when i moved the test setup 6XD plus 4x 3HC to the PNP Machine with only Power and CAN Bus Cables connected as before , it doesn't sync at all . The only thing that i changed is the length of the cable from 6XD to 3HC1( from 0.5 ft (shielded ) to 10 ft(shielded) .

Almost certainly the 10ft cable is faulty. Have you checked that it doesn't have crossed connections between the two ends?

@developeralgo222 said in Duet 3HC Expansion looses Connection:

After the test setup , moved everything to PnP Machine. As indicated above i tested each individual expansion 3HC board( 4 boards in total ) with nothing connected to them only power.
6XD --> Board 1 (with CAN Jumpers) CAN_IN / CAN_OUT -- Sync
6XD --> Board 2 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync
6XD --> Board 3 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync
6XD --> Board 4 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

Did you use the same cable between the 6XD and the 3HC in all cases?

Used same cables except 6XD to 3HC1 where i used a longer cable 10ft

[infiniteloop]

@developeralgo222

Termination on each board in the CAN Bus chain works ?

What @dc42 says. To test, throw-in some cheapo telephone lines (RJ11) and terminate CAN properly.

[developeralgo222]

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

Termination on each board in the CAN Bus chain works ?

What @dc42 says. To test, throw-in some cheapo telephone lines (RJ11) and terminate CAN properly.

i will try another cable , probably a phone cable or another ADSL 2+ high speed cable to replace the 10 ft cable from 6XD to 3HC1 and see if that makes a difference .

[infiniteloop]

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i will try another cable

Great

[developeralgo222]

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i will try another cable

Great

Made 2 different straight cables which are shielded and 1 std phone cable (Normally crossed, so i cut one end and crimped it to straight cable ).

Cable 1 - Made from CAT 6 Cable (using only 2 pairs for CAN Bus --technically on 2 wires are used) and RJ11 at the ends
Cable 2 - Made from CAT 8 Cable (using only 2 pairs for CAN Bus --technically on 2 wires are used) and RJ11 at the ends
Cable 3 - Made from Std Phone Cable (using only 2 pairs for CAN Bus --technically on 2 wires are used) and RJ11 at the ends

All of them behaved the same when testing each of the 4 x 3HC boards with nothing connected to them only power.

6XD --> Board 1 (with CAN Jumpers) CAN_IN / CAN_OUT -- Sync

6XD --> Board 2 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

6XD --> Board 3 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

6XD --> Board 4 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

Not sure why CAN Bus would be this unreliable. For sure i checked the cables , tested them using a different cable tester ( Sperry ) this time to rule out anything. i don't think its the cables after this round of tests

[droftarts]

@developeralgo222 something to do with the mounting? You have them stacked, using what looks like electrically conductive standoffs. I don’t know if that connects the grounds of the boards at a point that causes problems, rather than at the PSUs. You could try powering all four boards (or all boards) from one PSU, see if that makes a difference.

Ian

[developeralgo222]

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 something to do with the mounting? You have them stacked, using what looks like electrically conductive standoffs. I don’t know if that connects the grounds of the boards at a point that causes problems, rather than at the PSUs. You could try powering all four boards (or all boards) from one PSU, see if that makes a difference.

Ian

Hmmm!!! i don't think so unless the boards themselves have Power rail circuit layer (copper) touching the mounting holes . That would be really really bad. When i did the Test setup they were separated (Not mounted) although still on Brass standoffs. Your thought might be something to investigate , i will check that and try Nylon standoffs

[infiniteloop]

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Not sure why CAN Bus would be this unreliable.

It's not about CAN - I've built several applications for constructing machinery from the ground up, connecting controllers via CAN, and these connections are rock-solid. This said, I am now convinced that we can rule out your CAN wiring as cause of the problem. Which leads me to believe we might face an effect which I described as "interesting" early on: some interference between GND and Earth.

In your test setup, the boards were strictly isolated from Earth (which was just connected to the two PSUs). Earth is received from the electrical outlets and internally connected with the metal cases of the PSUs. In addition, the metal parts of you PnP machine should be "earthed" as well. The one million dollar question now is: have GND and Earth any potential contact? Or are they maybe decisively synced somewhere?

The latter case is something I've done myself, but is not without pitfalls, so I'd like to ask @droftarts for a second opinion. At least, he's on the same trail …