Duet 3HC Expansion looses Connection

developeralgo222

Currently i have moved All my Duets ----- Duet 6XD (has 2 external powered by another PSU drives so not consuming much current) and 3 x 3HC ---- onto a single 24V, 20A DC PSU.

i even made my own new RJ11 short cables to make sure they are correctly done and tested to be working correctly but all 3 x 3HC are not synchronizing with 6XD main board at all after upgrading to RRF 3.5.0_rc2

CAN ID 0 = 6XD ---- Input goes to CAN ID 1 -IN
CAN ID 1 = 3HC Board 1 --- CAN ID 1 OUT ---> CAN ID 2 IN
CAN ID 2 = 3HC Board 2 --- CAN ID 2 OUT ---> CAN ID 3 IN
CAN ID 3 = 3HC Board 3 with CAN Termination Jumpers in the chain

I have started testing by connecting one 3HC board at a time. Duets are really becoming a frustrating endeavor or a labor of love. arrggh!!!!. . i don't have Atmel ICE tool. Will also try a factory reset using CAN ID resets method

Duet 6XD has a red LED blinking normally
Board 1 - 3HC has continuous Red LED blinking
Board 2 & 3 ( 3HC ) both have red LED blinking 2 times followed by a slight pause then 3 times then a pause and then repeats with the following error

12/27/2023, 7:51:46 PM	Error: Expansion board 3 reconnected
12/27/2023, 7:51:45 PM	Error: Expansion board 2 reconnected
Error: Expansion board 2 reconnected

Tested connecting each board itself and they work ok until i connect all of them and they stop working. If i leave a jumper Board 2 and 3 , somehow they seem to sync . Not sure why that would be the case . when i remove jumpers on board 1 and board 2 and only leave the one of board 3 , they don't sync at all

i will now try factory reset On Duet 3 Expansion 3HC, by setting all four address switches off and board should reset its address and CAN bus speed to default and request firmware from the main board. Tried this and that did not work either ?

dc42

@developeralgo222 I am wondering whether boards 2 and 3 are continually resetting. If you bypass board 1, can you get a M122 report from board 2 or board 3?

Does it make any difference if you change the order of the boards, or swap two of them over? For example, swap boards 1 and 3.

Mr Yod

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i will now try factory reset On Duet 3 Expansion 3HC, by setting all four address switches off and board should reset its address and CAN bus speed to default and request firmware from the main board. Tried this and that did not work either ?

I was going to say "put all the firmware files on your mainboard and then set the switches of your expansion boards to 0000, one board connected at a time".

BTW: my RJ11 cable is just a 1.5m phone cable still bundled up from the packaging since it's too long, so length isn't a problem.
CAN Connect

After following dc42's advice, double-check physical connections: I have an ethernet connection that sometimes disconnects my Duet from the PC (the clip in the cable broke, so the connector can loosen up, so in your case maybe the connector doesn't fit perfectly due to a shim of plastic or something) and once a Z motor didn't turn because, by moving the connection from the mainboard to the expansion, a connection wasn't set properly.

developeralgo222

This post is deleted!

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I am wondering whether boards 2 and 3 are continually resetting. If you bypass board 1, can you get a M122 report from board 2 or board 3?

Does it make any difference if you change the order of the boards, or swap two of them over? For example, swap boards 1 and 3.

Yes i think they are resetting every 1 second

12/29/2023, 2:02:03 PM	Error: Expansion board 3 reconnected
Error: Expansion board 3 reconnected
12/29/2023, 2:02:02 PM	Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
12/29/2023, 2:02:02 PM	Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
12/29/2023, 2:02:01 PM	Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
12/29/2023, 2:02:01 PM	Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
12/29/2023, 2:02:00 PM

developeralgo222

@Mr-Yod said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i will now try factory reset On Duet 3 Expansion 3HC, by setting all four address switches off and board should reset its address and CAN bus speed to default and request firmware from the main board. Tried this and that did not work either ?

I was going to say "put all the firmware files on your mainboard and then set the switches of your expansion boards to 0000, one board connected at a time".

BTW: my RJ11 cable is just a 1.5m phone cable still bundled up from the packaging since it's too long, so length isn't a problem.

After following dc42's advice, double-check physical connections: I have an ethernet connection that sometimes disconnects my Duet from the PC (the clip in the cable broke, so the connector can loosen up, so in your case maybe the connector doesn't fit perfectly due to a shim of plastic or something) and once a Z motor didn't turn because, by moving the connection from the mainboard to the expansion, a connection wasn't set properly.

i guess if the CAN connectors on the Duet 3 boards are RJ11 female then if you have RJ11 Male connector it should not be an issue , if the cables are wired correctly and yes sometimes you have a loose connection. I checked all that and nothing is working to have them synced

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I am wondering whether boards 2 and 3 are continually resetting. If you bypass board 1, can you get a M122 report from board 2 or board 3?

Does it make any difference if you change the order of the boards, or swap two of them over? For example, swap boards 1 and 3.

i have swapped the boards, Added each one by one but to no avail. The only way its able to Sync but disconnecting sometimes , is by having CAN jumpers on Board 2 & 3 ( 3HC expansion boards ) and No CAN jumper on Board 1 (3HC ). No sure why that would work somehow ?

The only difference between Board 1 & ( 2 or 3) is that board 2 & 3 have SMC NPN Vacuum Pressure Sensors connected to the IO pins and they are powered from the 3HC VIN & GND as per your earlier post suggestion and also Board 3 has CAN Termination Jumpers

dc42

@developeralgo222 if the boards are resetting once a second then I need a M122 report from one of them in order to determine why. So try powering up the system, let it run for a few seconds, then unplug some CAN cables to bypass one or two of the 3HC boards; then get a M122 report from the remaining one. It's electrically safe to connect and disconnected CAN cables while the boards are powered.

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 if the boards are resetting once a second then I need a M122 report from one of them in order to determine why. So try powering up the system, let it run for a few seconds, then unplug some CAN cables to bypass one or two of the 3HC boards; then get a M122 report from the remaining one. It's electrically safe to connect and disconnected CAN cables while the boards are powered.

console_3HC1_But_both_3HC2-n-3HC3-with-CANTermJumpers.txt

console_Only_3HC1-with-CANTermJumpers_ByPassing_Both_3HC2_n_3HC3.txt

console_3HC1_n_3HC3-with-CANTermJumpers_Bypassing-3HC2.txt

console_3HC2_n_3HC3-with-CANTermJumpers_ByPassing-3HC1.txt

console_Only_3HC2-with-CANTermJumpers_ByPassing_Both_3HC1_n_3HC3.txt

console_Only_3HC3-with-CANTermJumpers_ByPassing_Both_3HC1_n_3HC2.txt

console_3HC1_3HC2_3HC3-with-CANTermJumpers.txt

NOTE: when i CAN connect only Duet 6XD to

3HC1 only with CAN Termination Jumpers on it (CAN ID 1 ) --- this works and synchronization occurs
3HC2 only with CAN Termination Jumpers on it (CAN ID 2 ) --- NO synchronization
3HC3 only with CAN Termination Jumpers on it (CAN ID 3 ) --- NO synchronization
3HC1 & 3HC2 only with CAN Termination Jumpers on 3HC2 --- NO synchronization
3HC1 & 3HC3 only with CAN Termination Jumpers on 3HC3 --- NO synchronization
3HC2 & 3HC3 only with CAN Termination Jumpers on 3HC3 --- NO synchronization
3HC1 & 3HC2 & 3HC3 with CAN Termination Jumpers on both 3HC2 and 3HC3 --- this works and synchronization occurs but there is an occasional timeout on 3HC1 . This should not be working as per Duet's documentation ???????

NOTE:

Also Duet 6XD, 3HC1 , 3HC2 & 3HC3 All share the same 24V , 20A DC Power Supply.
The only difference between 3HC1, 3HC2 & 3HC3 is that 3HC2 & 3HC3 each have 3 x SMC NPN Vacuum Pressure Digital Sensors -ZSE30A-01-N-L (Not Z-Probe) connected to them and share the Power VIN & GND Terminal on the Board itself as per @dc42 earlier suggestion

developeralgo222

@developeralgo222 @dc42 can you respond to this issue ?

droftarts

@developeralgo222 I know this sounds rather basic, but have you swapped the CAN wires around? It sounds like the cable from 3HC1 to 3HC2 isn’t working, or only partially, or picking up interference. Check continuity, and that resistance is within specification. How long are the cable runs?

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I know this sounds rather basic, but have you swapped the CAN wires around? It sounds like the cable from 3HC1 to 3HC2 isn’t working, or only partially, or picking up interference. Check continuity, and that resistance is within specification. How long are the cable runs?

Ian

I have checked the RJ11 cables and tested them with a Fluke professional cable tester to make sure they are ok. The lengths are :
6XD to 3HC1 —> 10 ft cable
3HC1 to 3HC2 & 3HC2 to 3HC3 —> 1ft cable

I am assuming that RJ11cable for CAN connection is supposed to be shielded Straight through cable ( ie wired at same pins positions on both ends ) as per the documentation . I was using high speed ADSL 2+ cable that were 10ft long , I tried also to shorten 2 of them to 1 ft but that did not solve the issue

developeralgo222

@developeralgo222 @dc42 & @droftarts Finally got this fixed . The issue was bad cable /connection from 6XD to Expansion Board 1 of 3HC . Never trust any cables from China , OMG when i decided to splice them , they are really crappy no shields and no proper insulations per pair. Buyer be aware.Have seen this in many chinese made products on Amazon.

droftarts

@developeralgo222 glad you got it fixed! It’s been bugging me what it could be if it wasn’t the cables!

Ian

developeralgo222

@droftarts

Got them powered up and syncing and then they started rapid flashing and lost synchronization again. so i unplugged all the extension boards and again started adding one by one, to see if any board or cable is the problem. I made all the connection cables and tested them for continuity myself to make sure they are ok .

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ? In my Case 6XD --> 3HC(1) --> 3HC2(2) --> 3HC(3 -- Has 2 CAN Bus jumpers)

This is getting to be a frustrating episode

droftarts

@developeralgo222 said in Duet 3HC Expansion looses Connection:

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ?

Yes, that should be correct. You should have termination at both ends of the bus.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ?

Yes, that should be correct. You should have termination at both ends of the bus.

Ian

i assume CAN Bus Termination Jumpers only on 3HC(3) but not on 6XD board ( i don't see any CAN Bus termination jumpers on 6XD ) in my case

droftarts

@developeralgo222 Yes, the 6XD should have CAN termination resistors built-in. See https://docs.duet3d.com/Duet3D_hardware/Duet_3_family/Duet_3_Mainboard_6XD_Hardware_Overview#can-fd-bus-expansion

There is a termination resistor fitted on each CAN bus, so normally this board must be at the end of the bus. There are drill-to-disconnect jumpers on the underside of the board that allow the termination resistor to be removed, however this is not required in normal operation.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 Yes, the 6XD should have CAN termination resistors built-in. See https://docs.duet3d.com/Duet3D_hardware/Duet_3_family/Duet_3_Mainboard_6XD_Hardware_Overview#can-fd-bus-expansion

There is a termination resistor fitted on each CAN bus, so normally this board must be at the end of the bus. There are drill-to-disconnect jumpers on the underside of the board that allow the termination resistor to be removed, however this is not required in normal operation.

Ian

Ok that's good. so it seems when i disconnect board 3 ( 3HC(3) ) still Boards (6XD, 3HC2(1) & 3HC(2) ) syncs even without a jumper on 3HC(2) but as soon as i connect board 3 (3HC(3) which has the jumpers it stops syncing.

A little troubleshooting :

6XD --> 3HC(1) ==> Syncs
6XD --> 3HC(1) ---> 3HC(2) with CAN Termination Jumpers) ==> Syncs
6XD --> 3HC(1) ---> 3HC(3) with CAN Termination Jumpers) ==> Syncs

Looses synchronization when i connect all 3 expansion boards ( 3HC(1) , 3HC(2) & 3HC(3) )

e.g 6XD --> 3HC(1) --> 3HC(2) with CAN Termination Jumpers) ==> Syncs



1/13/2024, 7:49:18 PM	M122 B2

1/13/2024, 7:49:18 PM	Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 184 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.1%,419) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:23:04 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21135, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21136, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21137, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -25/11, peak Rx sync delay 181, resyncs 9/0, no timer interrupt scheduled
VIN voltage: min 24.5, current 24.5, max 24.6
V12 voltage: min 12.3, current 12.4, max 12.4
MCU temperature: min 27.6C, current 36.0C, max 36.4C
Last sensors broadcast 0x00000000 found 0 203 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 7832, send timeouts 462, received 4244, lost 0, errs 4637716, boc 310, free buffers 38, min 38, error reg ff5f00
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0



1/13/2024, 7:49:11 PM	M122 B1

Diagnostics for board 1:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.0%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:22:57 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60248, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60248, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60249, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -17/13, peak Rx sync delay 183, resyncs 5/0, no timer interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.5
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 24.6C, current 32.3C, max 32.6C
Last sensors broadcast 0x00000000 found 0 163 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 10108, send timeouts 133, received 6112, lost 0, errs 14212604, boc 0, free buffers 38, min 38, error reg ff6500
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

Not sure what might be the issue. I don't think its the cables anymore, it has to be the Duet 3 3HC boards or the RRF firmware ?

dc42

@developeralgo222 are you quite certain that you have identified the CAN termination jumper positions correctly? The CAN bus is quite tolerant and will often work with a termination resistor missing or with an extra termination resistor, but probably not with two extra resistors.

Other things to check:

Do the contacts in the CAN sockets of board 2 look OK? Does the soldering of those sockets on the underside of the PCB look OK?
Are all 3 boards set to different CAN addresses?
Does 6XD - board 2 - board 3 work?