Duet 3HC Expansion looses Connection

developeralgo222

i need to have this swapped by Spool3D as soon as we can, if we can have a new one shipped express by tomorrow and i will express this defective 6XD to them express too. We have been troubleshooting this for over 4 weeks now and i am extremely behind on the PnP project. Should i contact Spool3D directly myself or one of Duet guys is going to get in touch with Spool3D today

Phaedrux

Please send an email to warranty@duet3d.com and CC your reseller. Include a link to this forum thread and the details of your original purchase. You'll receive a reply with a form to fill out.

developeralgo222

@Phaedrux said in Duet 3HC Expansion looses Connection:

Please send an email to warranty@duet3d.com and CC your reseller. Include a link to this forum thread and the details of your original purchase. You'll receive a reply with a form to fill out.

done

droftarts

@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:

On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.

So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:

On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.

So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.

Ian

I am not sure whether its the 6XD board itself or CAN Resistor termination. Either way it doesn't reliably work on Test Bench or PnP Machine. If its reliable and working correctly the Cable length should not matter on the Test bench or PnP Machine. Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .

The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:

On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.

So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.

Ian

I am not sure whether its the 6XD board itself or CAN Resistor termination. Either way it doesn't reliably work on Test Bench or PnP Machine. If its reliable and working correctly the Cable length should not matter on the Test bench or PnP Machine. Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .

The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.

developeralgo222

@developeralgo222

Please can someone from Duet Support Team that deals with Warranty (Warranty@duet3d.com) send me WAC code as required. Spool3D have got back to me but not Duet Team. i can't complete the required form without the required code from Duet. Time is of great importance to our project.

droftarts

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .

Wiggling the wires causes disconnections? Are you sure the pins in the RJ11 jack on the 6XD aren't bent or contaminated? It's not the soldering on the pins of the RJ11 on the back of the board, I can see that's fine. Though you could check for continuity between any of the pins on the RJ11 socket.

The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.

Most people won't ever need to touch these. If they do, they only need to drill out the via on the back of the board. If they decide they want to replace the termination, they just put a solder bridge back in. Far easier than trying to resolder two SMT resistors the size of a grain of sand!

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .

Wiggling the wires causes disconnections? Are you sure the pins in the RJ11 jack on the 6XD aren't bent or contaminated? It's not the soldering on the pins of the RJ11 on the back of the board, I can see that's fine. Though you could check for continuity between any of the pins on the RJ11 socket.

The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.

Yes i am sure of that. i have terminated all my network cables for the past 25 Years. I even decided that may be something was wrong with all my network cable termination on all my cables and bought Kenable ADSL 2+ cables that were recommended ( even though very expensive here in North America). Tested all Kenable ADSL 2+ cables and my own Cables for Continuity, Shorts, Miswires, Reversals/Crossed, Split pairs and Shielding with both Professional Fluke and Sperry Crimp and Test tools.

Even with Kenable ADLS 2+ it still did not work. So i can guarantee that its not the Cables

droftarts

@developeralgo222 I didn't say it was the cables. I said check the RJ11 socket on the 6XD.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I didn't say it was the cables. I said check the RJ11 socket on the 6XD.

Ian

I checked all the RJ11 Sockets and Pins to confirm during the troubleshooting. i was checking and double checking everything in the CAN Bus path because we couldn't find out what was causing the issue. Not sure why the RJ11 sockets would have an issue unless that's QA issue from the manufacturer of the RJ11 Sockets and most of those sockets are designed to take a real beating before they can have any fault in them and Yes, at this point it can be anything.

developeralgo222

@developeralgo222
@dc42, @droftarts @Phaedrux ,

Received the new 6XD board and started doing the Normal isolated Bench tests that i did earlier TEST1 and TEST 2 (Above) using Kenable ADSL 2+ Cables. Checked the resistors , they are OK

All boards have same Firmware RRF 3.5_rc3 with Expansion boards 3HC running 3.5_rc3 with Version 2.8 Bootloader

Self-made Cables are straight CAT 5 or CAT 6 twisted using 2 pairs with RJ11 termination

Same Test with 0.5m, 4m and 5m Where 6XD--> 0.5m(self-made)/4m(self-made)/5m(straight phone Cable) -->3HC1 -->0.5m(kenable)-->3HC2-->0.5m(kenable)-->3HC3-->0.5m(kenable)-->3HC4
Same Test with 15m Kenable ADSL 2+ cable Where 6XD-->15m(kenable)-->3HC1 -->0.5m(kenable)-->3HC2-->0.5m(kenable)-->3HC3--> 0.5m(kenable) -->3HC4

It does not look good so far, The tests are failing synchronization to the Expansion boards. i need to complete all the tests and then i will post the results.

T3P3Tony

@developeralgo222 are you testing with just 1 £HC on the bus to start with (and then test each 3HC individually)

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 are you testing with just 1 £HC on the bus to start with (and then test each 3HC individually)

Yes,

TEST 1: Completely Isolated Bench Test ---- Only CAN Bus & Power ( VIN & GND ) are connected and CAN Bus Termination Jumpers on the Last Expansion 3HC Board in that Chain

using Kenable ADSL 2+ (1m / 15m) Cable with RJ11 Termination

6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC1 ==== Do Not Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC2 ==== Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC3 ==== Do Not Sync
6XD ---> 1m/15m (Kenable ADSL 2+) --->3HC4 ==== Sync

using self-made 0.5m/4m/5m twisted CAT 5 & 6 Cable with RJ11 Termination

6XD ---> self-made 0.5m/4m/5m --->3HC1 ==== Do Not Sync
6XD ---> self-made 0.5m/4m/5m --->3HC2 ==== Sync
6XD ---> self-made 0.5m/4m/5m --->3HC3 ==== Do Not Sync
6XD ---> self-made 0.5m/4m/5m --->3HC4 ==== Sync

using standard straight Phone (4m) cable with RJ11 Termination

6XD ---> 4m (Phone Cable 4m) --->3HC1 ==== Do Not Sync
6XD ---> 4m (Phone Cable 4m) --->3HC2 ==== Sync
6XD ---> 4m (Phone Cable 4m) --->3HC3 ==== Sync
6XD ---> 4m (Phone Cable 4m) --->3HC4 ==== Sync

TEST 2: --being done now

using standard straight Phone cables 4m / 0.5m with RJ11 Termination

6XD ---> 4m ---> 3HC1 ---> 0.5m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Do Not Sync
6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync

2/2/2024, 9:40:03 AM

M122 B2
Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156544, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,175) HEAT(2,nWait 6,0.0%,136) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,64) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,408) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,259), total 100.0%
Owned mutexes:
Last reset 00:02:38 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 21202, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/4, peak Rx sync delay 182, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.3, current 12.4, max 12.4
MCU temperature: min 34.9C, current 35.6C, max 35.6C
Last sensors broadcast 0x00000000 found 0 113 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 882, send timeouts 0, received 540, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

2/2/2024, 9:39:41 AM

M122 B3
Diagnostics for board 3:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156544, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,175) HEAT(2,nWait 6,0.0%,125) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,64) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,404) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Owned mutexes:
Last reset 00:02:16 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11132, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11131, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min n/a, mspos 8, reads 11131, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -4/5, peak Rx sync delay 181, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.2
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 33.9C, current 34.2C, max 34.5C
Last sensors broadcast 0x00000000 found 0 183 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 726, send timeouts 0, received 443, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0



2/2/2024, 9:39:34 AM

M122 B4
Diagnostics for board 4:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.3 (2024-01-24 17:53:31)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156664, free system stack 202 words
Tasks: Move(3,nWait 7,0.0%,180) HEAT(2,nWait 6,0.0%,136) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,65) TMC(2,nWait 6,6.5%,94) MAIN(1,running,92.3%,402) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,264), total 100.0%
Owned mutexes:
Last reset 00:02:09 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3937, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3937, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 3936, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/5, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.3, current 24.3, max 24.3
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 33.4C, current 33.4C, max 33.8C
Last sensors broadcast 0x00000000 found 0 181 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 710, send timeouts 0, received 434, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

i have flashed 3HC1 bootloader and firmware (just in case its corrupted ) to see if that helps but 3HC1 does not seem to be able to sync.

@dc42 , @droftarts , @T3P3Tony

Is there anything i might be missing on Main 6XD board that might prevent 4 x 3HC boards from syncing ? Since the maximum number of CAN connected boards (in RRF 3.5 ) is 20 for 6XD and that's within the Limit to allow connection of 4 x 3HC boards.

What does Standalone Mode mean in the diagnostic print out M122 B0 . Is there a different mode ?

RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.3 (2024-01-24 17:59:29) running on Duet 3 MB6XD v1.01 or later (standalone mode)

soare0

@developeralgo222
My 2c, if allowed, as I followed the topic a little.
I only have a 6XD with 1x3HC, and they work flawlesly, and I am sure 3 of them would, if ok.
I noticed that you are not a hardware engineer, so it is understandable your... technique is not... up to standards, but yet maybe I can help.

For such connections length, termination, shielding, twisting, is almost irrelevant, excepting the case that you would have nearby some very powerful EMI sources.
The worst thing happened was not having common ground, I smiled when noticed, bot also I saw you corrected. It is a must when you work with multiple power supply (not talking about isolation needs, when it is the case, but this is not), as we can easily have a big common mode voltage, wich may be big enough in such cases, either LF, 50Hz, or HF... xxx Hz. So, a big wire between two V- (one of each PS), is the first thing needed to be taking care of, prior to the first power up. After that, connect all the stuff exactly as you did.
Note: voltage differential may cripple some stuff on some boards, if that connexion is not in place, it is simple hazard most of the times.
CAN is reliable enough, do not worry about it. Common mode noise/bias is another thing, and this may be troubling. But not CAN itself, as it compensates for most of this stuff, anyway.
I used simple telephone cable, untwisted, unshielded, and ... no problem. Even crimped myself, with a very doubtful crimping tool. Just take care to crimp hard enough... You will NOT, I repeat, you will NOT have problems with this, there is not special cable for CAN, it is a simple serial connection, with hardware/firmware specific communication protocols.
Just dissasemble all, and redo the test platform, as it were when working. Test each 3HC with 6XD separately, with the same cable. Then whatever one of them, changing cables. Then add the second,, with another (tested) cable, Then add the third, with the third cable. This way you will be able to separate problems.
If all works reliable, put the 6XD in place, and test it. Add just the first board with the first cable, and then build up until the third, retesting at each step.
Take care that you will want to repeatedly change the adresses of the boards, when testing like this, at least when testing each board, to check if each board works well on each address it would eventually be connected.
Build only step by step, even in that ... structure of yours. I very much suspect there is a problem with those metal ..... whatever, or with a cable with a loose connection. I would have the brass risers replaced by some plastic ones, even if, normally it should not matter. Mine are, I think, plated copper/brass, I need to look, maybe in some paranoia excess I changed to PE ones. Again, it should not matter, but again, maybe there is also some groove there, etc...
Never assume it is somebody else fault (bad board, bad configuration, etc), until you can prove it. I am sorry to be blunt, but it is one of the main mistakes you did here, let name it the first. I know the drill, as I observed it SO MANY times in the past. Usually, the mechanic engineer and the electronics engineer, blame each other (most of the times the mechanics blames electronists, and most of the times they are wrong).
So the key to debug this is to check yourself at every step, do it as gradually as possible. I know this is a test engineer behaviour, wich I hardly learn from the best...
The second mistake is that you tried to test all the system at once, without proper tools and knowledge. Now, when it works, it works, hehe, luck is luck...
The third mistake is to assume that you really know what you are doing. I did it many times, too. That wire, man, that wire again...!

Good luck...

Phaedrux

@developeralgo222 said in Duet 3HC Expansion looses Connection:

What does Standalone Mode mean in the diagnostic print out M122 B0 . Is there a different mode ?

Standalone means without a raspberry Pi.

T3P3Tony

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Is there anything i might be missing on Main 6XD board that might prevent 4 x 3HC boards from syncing

No, 4x3HCs should work.

Looks like it was B1 that was causing the issues

developeralgo222

@T3P3Tony said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Is there anything i might be missing on Main 6XD board that might prevent 4 x 3HC boards from syncing

No, 4x3HCs should work.

Looks like it was B1 that was causing the issues.

Is there an easier way to identify if the issue is the 3HC1 Board itself or the CAN Bus connectors on 3HC1 board. If the issue is the 3HC Board itself then i would like to find out and have it swapped.

Has anyone in the Duet Forum actually tested something similar to what i have configured

6XD MainBoard with 4 x 3HC boards

Please any pointers that i have not done yet ?

I am able to get it to Sync with 1 , 2 or 3 Duet 3HC boards (Even though not very reliable ) but i can't get it work with 4 x 3HC boards. I don't believe that cables are the issue. Kenable ADSL 2+ Cables which are recommended do not work even on the test Bench setup that seems to sync when using a phone cable (4m) .

6XD ---> 4m ---> 3HC2 ---> 0.5m ---> 3HC3 ---> 0.5m ---> 3HC4 ==== Sync

Phaedrux

If you leave 3hc1 disconnected are you able to get all the other boards synced with any cable combo?

T3P3Tony

@developeralgo222 What happens if you do the following.

Connect the 6XD to 3HC Board 2, via a known good cable. confirm sync and communications
Unplug the CAN cable at the 3HC end. plug in Board 1, power it up, send M122 B1