Duet 3HC Expansion looses Connection

developeralgo222

@gloomyandy said in Duet 3HC Expansion looses Connection:

@developeralgo222 Have you tested the long cable to just a single 3HC board (which must be the one with the terminator enabled - I assume the others do not have it enabled)?

Yes the Terminator jumpers are only on 3HC4 board. Just tested the long 15m Cable from 6XD --> 3HC4 only ( 2 boards ) and does not sync

M122 B4
CAN response timeout: board 4, req type 6024, RID 84

infiniteloop

@developeralgo222 Thank you for running all these tests and collecting the data. I’m not qualified to interpret the results in detail, but as you’ve eliminated all other potential suspects, it is now crystal clear that CAN is the single possible culprit. In my eyes, a great achievement and a major step towards a solution. Let's see what idea @DC42 comes up with. Maybe he wants you to run some more tests to pin down the root cause, so please keep up your test environment for the time being.

droftarts

@developeralgo222 I'm wondering if the CAN termination is damaged on the 6XD, as there appears to be little consistency between connections with the 3HC boards. I doubt you have four bad 3HCs!

The circuit is very similar there compared to the 3HC:

The CAN 1 termination resistors are R133 and R134, but aren't marked on the silkscreen I think. They are behind the CAN 1 OUT RJ11 connector. Also check that the drill-out termination vias (CAN1_TERM) on the back of the board are still intact:

Do you have another mainboard you can test? I think if you did you might have already tried that by now...

If that doesn't help, I think we're at the point where we should swap out your 6XD under warranty.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I'm wondering if the CAN termination is damaged on the 6XD, as there appears to be little consistency between connections with the 3HC boards. I doubt you have four bad 3HCs!

The circuit is very similar there compared to the 3HC:

The CAN 1 termination resistors are R133 and R134, but aren't marked on the silkscreen I think. They are behind the CAN 1 OUT RJ11 connector. Also check that the drill-out termination vias (CAN1_TERM) on the back of the board are still intact:

Do you have another mainboard you can test? I think if you did you might have already tried that by now...

If that doesn't help, I think we're at the point where we should swap out your 6XD under warranty.

Ian
I don’t have another 6XD that I can use for test. What’s the process of swapping the 6XD , do I send to spool3D ( where I bought it) or to Duet in UK ?

droftarts

@developeralgo222 said in Duet 3HC Expansion looses Connection:

I don’t have another 6XD that I can use for test. What’s the process of swapping the 6XD , do I send to spool3D ( where I bought it) or to Duet in UK ?

We'll get Spool3D to swap it out for you. Can you confirm that you've checked the CAN resistors on the 6XD?

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

I don’t have another 6XD that I can use for test. What’s the process of swapping the 6XD , do I send to spool3D ( where I bought it) or to Duet in UK ?

We'll get Spool3D to swap it out for you. Can you confirm that you've checked the CAN resistors on the 6XD?

Ian

Yes CAN termination resistors at the back looked ok when i tested the CAN_H and CAN_L pins (118 Ohms ) but by closely looking at them on the back i don't see any actual 2 x SMT Resistors (R133 = 59 Ohms, R134 = 59 Ohms ) on the solder pads other than the pads being connected as you can see on the photo.

developeralgo222

@developeralgo222

i need to have this swapped by Spool3D as soon as we can, if we can have a new one shipped express by tomorrow and i will express this defective 6XD to them express too. We have been troubleshooting this for over 4 weeks now and i am extremely behind on the PnP project. Should i contact Spool3D directly myself or one of Duet guys is going to get in touch with Spool3D today

Phaedrux

Please send an email to warranty@duet3d.com and CC your reseller. Include a link to this forum thread and the details of your original purchase. You'll receive a reply with a form to fill out.

developeralgo222

@Phaedrux said in Duet 3HC Expansion looses Connection:

Please send an email to warranty@duet3d.com and CC your reseller. Include a link to this forum thread and the details of your original purchase. You'll receive a reply with a form to fill out.

done

droftarts

@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:

On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.

So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:

On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.

So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.

Ian

I am not sure whether its the 6XD board itself or CAN Resistor termination. Either way it doesn't reliably work on Test Bench or PnP Machine. If its reliable and working correctly the Cable length should not matter on the Test bench or PnP Machine. Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .

The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:

On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.

So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.

Ian

I am not sure whether its the 6XD board itself or CAN Resistor termination. Either way it doesn't reliably work on Test Bench or PnP Machine. If its reliable and working correctly the Cable length should not matter on the Test bench or PnP Machine. Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .

The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.

developeralgo222

@developeralgo222

Please can someone from Duet Support Team that deals with Warranty (Warranty@duet3d.com) send me WAC code as required. Spool3D have got back to me but not Duet Team. i can't complete the required form without the required code from Duet. Time is of great importance to our project.

droftarts

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .

Wiggling the wires causes disconnections? Are you sure the pins in the RJ11 jack on the 6XD aren't bent or contaminated? It's not the soldering on the pins of the RJ11 on the back of the board, I can see that's fine. Though you could check for continuity between any of the pins on the RJ11 socket.

The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.

Most people won't ever need to touch these. If they do, they only need to drill out the via on the back of the board. If they decide they want to replace the termination, they just put a solder bridge back in. Far easier than trying to resolder two SMT resistors the size of a grain of sand!

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .

Wiggling the wires causes disconnections? Are you sure the pins in the RJ11 jack on the 6XD aren't bent or contaminated? It's not the soldering on the pins of the RJ11 on the back of the board, I can see that's fine. Though you could check for continuity between any of the pins on the RJ11 socket.

The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.

Yes i am sure of that. i have terminated all my network cables for the past 25 Years. I even decided that may be something was wrong with all my network cable termination on all my cables and bought Kenable ADSL 2+ cables that were recommended ( even though very expensive here in North America). Tested all Kenable ADSL 2+ cables and my own Cables for Continuity, Shorts, Miswires, Reversals/Crossed, Split pairs and Shielding with both Professional Fluke and Sperry Crimp and Test tools.

Even with Kenable ADLS 2+ it still did not work. So i can guarantee that its not the Cables

droftarts

@developeralgo222 I didn't say it was the cables. I said check the RJ11 socket on the 6XD.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I didn't say it was the cables. I said check the RJ11 socket on the 6XD.

Ian

I checked all the RJ11 Sockets and Pins to confirm during the troubleshooting. i was checking and double checking everything in the CAN Bus path because we couldn't find out what was causing the issue. Not sure why the RJ11 sockets would have an issue unless that's QA issue from the manufacturer of the RJ11 Sockets and most of those sockets are designed to take a real beating before they can have any fault in them and Yes, at this point it can be anything.

developeralgo222

@developeralgo222
@dc42, @droftarts @Phaedrux ,

Received the new 6XD board and started doing the Normal isolated Bench tests that i did earlier TEST1 and TEST 2 (Above) using Kenable ADSL 2+ Cables. Checked the resistors , they are OK

All boards have same Firmware RRF 3.5_rc3 with Expansion boards 3HC running 3.5_rc3 with Version 2.8 Bootloader

Self-made Cables are straight CAT 5 or CAT 6 twisted using 2 pairs with RJ11 termination

Same Test with 0.5m, 4m and 5m Where 6XD--> 0.5m(self-made)/4m(self-made)/5m(straight phone Cable) -->3HC1 -->0.5m(kenable)-->3HC2-->0.5m(kenable)-->3HC3-->0.5m(kenable)-->3HC4
Same Test with 15m Kenable ADSL 2+ cable Where 6XD-->15m(kenable)-->3HC1 -->0.5m(kenable)-->3HC2-->0.5m(kenable)-->3HC3--> 0.5m(kenable) -->3HC4

It does not look good so far, The tests are failing synchronization to the Expansion boards. i need to complete all the tests and then i will post the results.

T3P3Tony

@developeralgo222 are you testing with just 1 £HC on the bus to start with (and then test each 3HC individually)

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 are you testing with just 1 £HC on the bus to start with (and then test each 3HC individually)

Yes,

TEST 1: Completely Isolated Bench Test ---- Only CAN Bus & Power ( VIN & GND ) are connected and CAN Bus Termination Jumpers on the Last Expansion 3HC Board in that Chain

using Kenable ADSL 2+ (1m / 15m) Cable with RJ11 Termination

6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC1 ==== Do Not Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC2 ==== Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC3 ==== Do Not Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC4 ==== Sync

using self-made 0.5m/4m/5m twisted CAT 5 & 6 Cable with RJ11 Termination

6XD ---> self-made 0.5m/4m/5m --->3HC1 ==== Do Not Sync
6XD ---> self-made 0.5m/4m/5m --->3HC2 ==== Sync
6XD ---> self-made 0.5m/4m/5m --->3HC3 ==== Do Not Sync
6XD ---> self-made 0.5m/4m/5m --->3HC4 ==== Sync

using standard straight Phone (4m) cable with RJ11 Termination

6XD ---> 4m (Phone Cable 4m) --->3HC1 ==== Do Not Sync
6XD ---> 4m (Phone Cable 4m) --->3HC2 ==== Sync
6XD ---> 4m (Phone Cable 4m) --->3HC3 ==== Sync
6XD ---> 4m (Phone Cable 4m) --->3HC4 ==== Sync

TEST 2: --being done now

using standard straight Phone cables 4m / 0.5m with RJ11 Termination

6XD ---> 4m ---> 3HC1 ---> 0.5m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Do Not Sync
6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync

2/2/2024, 9:40:03 AM

M122 B2
Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156544, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,175) HEAT(2,nWait 6,0.0%,136) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,64) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,408) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,259), total 100.0%
Owned mutexes:
Last reset 00:02:38 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/4, peak Rx sync delay 182, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.3, current 12.4, max 12.4
MCU temperature: min 34.9C, current 35.6C, max 35.6C
Last sensors broadcast 0x00000000 found 0 113 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 882, send timeouts 0, received 540, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

2/2/2024, 9:39:41 AM

M122 B3
Diagnostics for board 3:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156544, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,175) HEAT(2,nWait 6,0.0%,125) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,64) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,404) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Owned mutexes:
Last reset 00:02:16 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11132, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11131, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11131, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -4/5, peak Rx sync delay 181, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 33.9C, current 34.2C, max 34.5C
Last sensors broadcast 0x00000000 found 0 183 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 726, send timeouts 0, received 443, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0



2/2/2024, 9:39:34 AM

M122 B4
Diagnostics for board 4:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156664, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,180) HEAT(2,nWait 6,0.0%,136) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,65) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,402) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,264), total 100.0%
Owned mutexes:
Last reset 00:02:09 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3937, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3937, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3936, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/5, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.3, current 24.3, max 24.3
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 33.4C, current 33.4C, max 33.8C
Last sensors broadcast 0x00000000 found 0 181 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 710, send timeouts 0, received 434, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

i have flashed 3HC1 bootloader and firmware (just in case its corrupted ) to see if that helps but 3HC1 does not seem to be able to sync.

@dc42 , @droftarts , @T3P3Tony

Is there anything i might be missing on Main 6XD board that might prevent 4 x 3HC boards from syncing ? Since the maximum number of CAN connected boards (in RRF 3.5 ) is 20 for 6XD and that's within the Limit to allow connection of 4 x 3HC boards.

What does Standalone Mode mean in the diagnostic print out M122 B0 . Is there a different mode ?

RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.3 (2024-01-24 17:59:29) running on Duet 3 MB6XD v1.01 or later (standalone mode)