Duet 3HC Expansion looses Connection

dc42

@developeralgo222 have you confirmed that the wires are not crossed between the ends of the 10ft cable? Your cable tester might not detect that. See https://docs.duet3d.com/en/User_manual/Machine_configuration/CAN_connection#example-of-a-good-cable and the photo after that.

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 have you confirmed that the wires are not crossed between the ends of the 10ft cable? Your cable tester might not detect that. See https://docs.duet3d.com/en/User_manual/Machine_configuration/CAN_connection#example-of-a-good-cable and the photo after that.

Yes i double checked that . i was suspecting that it might be the case but no. Cable tester that i use checks for all kinds of errors on cable from crossed , shorts , reversed, miswires, continuity, shield , etcs i have used various Fluke professional Cable testers for over 20 years and they have never failed, so i am confident when after all testing it says the cable is good, then its good.

They are Straight through cable (Left to right ) and on both ends

Blue, Blue-white, Orange, Orange-white

Technically , in this case , Blue-White and Orange wires are used by CAN Bus connection in Duet Boards

droftarts

@developeralgo222 i don’t know if this matters that much for a short cable, but wires of the same colour, ie blue and white-with-blue-stripe, make up the twisted pair. These should be wired to the centre two pins that Duet use for CAN. The other pair goes on the pins outside these. You’re using one wire from each twisted pair.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 i don’t know if this matters that much for a short cable, but wires of the same colour, ie blue and white-with-blue-stripe, make up the twisted pair. These should be wired to the centre two pins that Duet use for CAN. The other pair goes on the pins outside these. You’re using one wire from each twisted pair.

Ian

Let me try it . i have tried almost everything Grounding, and everything in between that has been suggested but to no avail. Sometimes its usually the smallest thing that creates a nightmare.

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 i don’t know if this matters that much for a short cable, but wires of the same colour, ie blue and white-with-blue-stripe, make up the twisted pair. These should be wired to the centre two pins that Duet use for CAN. The other pair goes on the pins outside these. You’re using one wire from each twisted pair.

Ian

Let me try it . i have tried almost everything Grounding, and everything in between that has been suggested but to no avail. Sometimes its usually the smallest thing that creates a nightmare.

Didn't make a difference at all .

I have tried every kind of Cable connection as per the requirement but there is no reliable sync of CAN connection. I don't believe i am the only one who is building a PnP or a 3D printer that is encountering this issue. i have worked with various kinds of connections and boards but have never encountered such an issue with CAN Bus.

Can't Sync 3HC boards no matter the cable. Have used the documentation to double check everything. Remake the cables and test them. Not sure if it might actually be a Duet 3 3HC Hardware issue that is causing this ?

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Twisted pair cables terminated in RJ11 connectors are sold in some countries as "High Speed ADSL cables". One supplier of such cables is Kenable. Note, ADSL and telephone cables made with flat wire often cross the connections, making them unsuitable

The price of kenable cables in US is so expensive while in UK they are extremely cheap. Can anyone point me to a link on Amazon or ebay for cable that works with Duet 3HC or 6XD CAN Bus without me cutting and remaking it, and is affordable

droftarts

@developeralgo222 I'm sorry you're having so many problems with this. It really shouldn't be this difficult, and it has us foxed as well! Maybe the best option is to return to the last known working configuration, which was as a bench setup.

For cables in the USA on Amazon, these look like the correct ones: https://www.amazon.com/Kenable-Speed-Broadband-Modem-Cable/dp/B011SS7LYU

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I'm sorry you're having so many problems with this. It really shouldn't be this difficult, and it has us foxed as well! Maybe the best option is to return to the last known working configuration, which was as a bench setup.

For cables in the USA on Amazon, these look like the correct ones: https://www.amazon.com/Kenable-Speed-Broadband-Modem-Cable/dp/B011SS7LYU

Ian

I agree this should not be this difficult. I have checked and double checked a lot of things but i can't get the CAN Sync to work. I decided to close my eyes, curse as much as i can and then ordered those cables let's hope that those help. Will get them in a week.

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I'm sorry you're having so many problems with this. It really shouldn't be this difficult, and it has us foxed as well! Maybe the best option is to return to the last known working configuration, which was as a bench setup.

For cables in the USA on Amazon, these look like the correct ones: https://www.amazon.com/Kenable-Speed-Broadband-Modem-Cable/dp/B011SS7LYU

Ian

I agree this should not be this difficult. I have checked and double checked a lot of things but i can't get the CAN Sync to work. I decided to close my eyes, curse as much as i can and then ordered those cables let's hope that those help. Will get them in a week.

@dc42, @droftarts , What is the expected terminating resistors values at the end of the CAN Bus for Duet boards? Ideally , Each resistor should ideally be 120 ohm, for a 60 ohm difference between CANH and CANL. When i measure the resistance with Multimeter on the terminated 3HC Board i get a difference of 70.4 Ohms and for Unterminated CAN_H & CAN_L i get 126.6 Ohms ? are this the expected values ?

Was doing some research and to continue troubleshooting, came across some explanation
General CAN Bus wiring and troubleshooting concepts document

Other CAN Bus terminating resistor values can be used and calculated based on specific baudrates and bus lengths - but, as you increase the baudrate, the pickier it gets about correct termination, whereas on lower baudrates you can get away with missing terminating resistors, by chance

Looking at the Duet Board / Schematic - i see Shield_GND terminal on the main board 6XD which i assume is for Ethernet Grounding.

Where is the CAN Grounding on Duet Boards ? since The CAN_GND of the CAN network should be connected to the functional earth potential (FE) at only one point i.e either on Main Board ( 6XD ) or on the Last 3HC on the Chain . Looking at the Kicad PCB design for 6XD , i see CAN_SHD_GND is that the same as CAN_GND in Duet 6XD design ?

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

Is there anything I might be forgetting on this GND solution. I am trying to avoid creating GND loops

That’s complex. As a general rule, imagine a star topology, I.e. for each and every voltage, there should be a single point (or rail) from where to further route the power lines.

For your dual PSUs, you already have this central point: V+ (1 rail with 3 connectors on each PSU) is now interconnected as discussed above in this thread.

Now, if you connect GND (of the DC-output) to Earth, do it at a single point as well. Think of it as a common rail where all GND lines meet - kind of a ”local Earth”. Don’t take GND or V– from distant parts of the frame. Aside from the common rail, consider them as not being the same.

Voltage lines may be split or chained on their run, but may then never refer to another source of the same voltage again (that’s a loop). You often run into this trap with shielded lines: the shield carries GND (or, in some cases, V–), but if you supply GND from both ends, you might involuntarily establish a ground loop. The golden rule here is to always feed GND from the origin - leave it unconnected at the destination end.

The fact that most devices in your machine are controlled via V– complicates things. All switched V– lines from the DUETs may never have contact with the ”local Earth”, GND or any other V–.

Grounding the PnP machine is another topic: not all metal parts are always in touch with each other, there are potential isolators such as coatings, belts or lubricants in the way, some parts are clearly isolated by design (printed joints etc). Short cable bridges help in some cases, but for moving parts, you might need to route a dedicated GND line through your cable chain.

Considering USB ground loops, I recommend this article from the documentation: USB ground loops

@infiniteloop , @dc42 , @droftarts

Can you give detail on this ? Currently i have the 2 PSUs connected like this

PSUs_Negative_Output_Connection_With_DC_to_AC_GND_1.png

Do i need to connect anything differently here ?

infiniteloop

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Can you give detail on this ? Currently i have the 2 PSUs connected like this

I’d rather like to take the Monty Python turn: And now for something completely different …

Well, let’s see where we are:

your wiring of the PSUs looks OK.
the test setup for CAN with isolated boards works - am I right with this?
if mounted on the PnP machine, only the CAN connection 6XD <-> 3HC „1“ works, all other connections fail - am I right with this?

If this is the current state of affairs, I’d like you to run another test: As of the photo below, the boards are stacked with electrically conductive standoffs who are, in turn, attached to the frame of the PnP machine. Now, can you detach the whole stack from the machine - but keep the standoffs between the boards in place? I’d like to know whether CAN becomes dysfunctional due to the GND it gets from the PnP machine or due to the electrical interconnection of the boards via the standoffs alone.

Please be assured that me, like @droftarts, am sorry for the problems you face. More so, I’m sincerely foxed by my inability to spot the root cause of the mystic CAN bus failure.

droftarts

@infiniteloop The stack of boards in that image are standing on what looks like a plastic toolbox, not connected to the machine.

As far as I can tell, there is no copper in close proximity to the mounting holes, at least on the 3HC v1.0 I have in my hands. There's no exposed copper around the holes, no via through-plating, and it looks like the copper ground plane doesn't extend to the holes. So I'm pretty sure they are electrically isolated from each other.

@developeralgo222 Plugging the two power supplies in to the same power bar should be fine. While we don't advise the same thing to get around USB ground loops, putting the two earths as close to each other as possible should largely mitigate a ground loop, if the PSUs are similar. I don't know if there's something about the phasing between them causing a problem; I'm no electrical engineer!

However... to mitigate that, have you tried powering all boards from one PSU, at least for testing? One PSU should have plenty of current to power all the boards, so long as no demanding load is put on it. Boards usually draw around 200-300mA with no peripherals connected.

For the CAN question, I don't think termination resistors values need changing, because we (a) run at 1Mbit (ie not that fast) and (b) your CAN bus is short (less than five meters). The 70.4 Ohm reading on the terminated 3HC might be interesting; can you check the resistance in the same way on all the boards, and see if there has been damage to the R37 and R38? These are two tiny 59R resistors (I think they work in series to be around 120R) between the CAN RJ11 ports and the CAN termination jumpers. Highlighted red (from the ibom here) :

This is the circuit diagram:

Though it may just be a measurement error.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@infiniteloop The stack of boards in that image are standing on what looks like a plastic toolbox, not connected to the machine.

As far as I can tell, there is no copper in close proximity to the mounting holes, at least on the 3HC v1.0 I have in my hands. There's no exposed copper around the holes, no via through-plating, and it looks like the copper ground plane doesn't extend to the holes. So I'm pretty sure they are electrically isolated from each other.

@developeralgo222 Plugging the two power supplies in to the same power bar should be fine. While we don't advise the same thing to get around USB ground loops, putting the two earths as close to each other as possible should largely mitigate a ground loop, if the PSUs are similar. I don't know if there's something about the phasing between them causing a problem; I'm no electrical engineer!

However... to mitigate that, have you tried powering all boards from one PSU, at least for testing? One PSU should have plenty of current to power all the boards, so long as no demanding load is put on it. Boards usually draw around 200-300mA with no peripherals connected.

For the CAN question, I don't think termination resistors values need changing, because we (a) run at 1Mbit (ie not that fast) and (b) your CAN bus is short (less than five meters). The 70.4 Ohm reading on the terminated 3HC might be interesting; can you check the resistance in the same way on all the boards, and see if there has been damage to the R37 and R38? These are two tiny 59R resistors (I think they work in series to be around 120R) between the CAN RJ11 ports and the CAN termination jumpers. Highlighted red (from the ibom here) :

This is the circuit diagram:

Though it may just be a measurement error.

Ian

When i measure resistance across unterminated 3HC boards CAN_H & CAN_L i get 140 Ohms for Board 1, 2, 3 and then Board 4 is 70 Ohms which is correct because its terminated and that's the differential resistance. Early in my troubleshooting phase, i did try to power all from one PSU but that did not make a difference. i will try again for testing purpose .

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Can you give detail on this ? Currently i have the 2 PSUs connected like this

I’d rather like to take the Monty Python turn: And now for something completely different …

Well, let’s see where we are:

your wiring of the PSUs looks OK.

the test setup for CAN with isolated boards works - am I right with this?

if mounted on the PnP machine, only the CAN connection 6XD <-> 3HC „1“ works, all other connections fail - am I right with this?

If this is the current state of affairs, I’d like you to run another test: As of the photo below, the boards are stacked with electrically conductive standoffs who are, in turn, attached to the frame of the PnP machine. Now, can you detach the whole stack from the machine - but keep the standoffs between the boards in place? I’d like to know whether CAN becomes dysfunctional due to the GND it gets from the PnP machine or due to the electrical interconnection of the boards via the standoffs alone.

Please be assured that me, like @droftarts, am sorry for the problems you face. More so, I’m sincerely foxed by my inability to spot the root cause of the mystic CAN bus failure.

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards. They no longer touch the frame or any metal attached to the frame.

infiniteloop

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards.

I propose to use this setup (isolated stack, mounted on the PnP) for further tests:

follow @droftarts' proposal and use just a single PSU
with all peripherals detached, just VIN and CAN connected, are all boards recognised?
if not, use the short CAN cables from your previous test setup.

At least the results of the last test should match those from your previous test setup on the bench: all boards should connect. If not, there must be a difference we are not aware of. Let's see, please share the results.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards.

I propose to use this setup (isolated stack, mounted on the PnP) for further tests:

follow @droftarts' proposal and use just a single PSU

with all peripherals detached, just VIN and CAN connected, are all boards recognised?

if not, use the short CAN cables from your previous test setup.

At least the results of the last test should match those from your previous test setup on the bench: all boards should connect. If not, there must be a difference we are not aware of. Let's see, please share the results.

For testing & Troubleshooting

i have connected all the boards ( 6XD plus 4x 3HC ) to a single DC PSU ( 24V , 20A )
Only Power ( VIN & GND ) and CAN Bus are connected.
i used a short standard Phone cable between 6XD & 3HC1 (that i cut to a straight Cable since phone cable are crossed)

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Test results:

All boards sync lights are synchronized but i get the following errors

1/26/2024, 5:48:48 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:48:06 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:47:04 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:23:04 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:14:12 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:13:07 PM	Error: Expansion board 1 stopped sending status

i think there might be PSU issue plus CAN cable issue, Once i get the recommended Kenable ADSL 2+ cables i should be able to completely rule out CAN Cable issue.

Even with a ray of hope it still doesn't work as expected

infiniteloop

@developeralgo222

Error: Expansion board 1 stopped sending status

That's one for @droftarts or @dc42, I am unable to draw any qualified conclusions from that. Hm, what's special about board #1? Is this by chance the same board which you, on January 19, reported as being the only one that synchronises with the 6XD?

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Yes, you mentioned earlier the delivery could take a week or so. Maybe that with these, the results of the initial test setup can be reproduced - prove me wrong, but I think the greater distance between the 6XD and the stack now is the only difference to the initial bench test.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

Error: Expansion board 1 stopped sending status

That's one for @droftarts or @dc42, I am unable to draw any qualified conclusions from that. Hm, what's special about board #1? Is this by chance the same board which you, on January 19, reported as being the only one that synchronises with the 6XD?

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Yes, you mentioned earlier the delivery could take a week or so. Maybe that with these, the results of the initial test setup can be reproduced - prove me wrong, but I think the greater distance between the 6XD and the stack now is the only difference to the initial bench test.

Yes it's the same 3HC board. I think the Cable from 6XD to 3HC1 might be having issues too. Once the new Kenable ADSL 2+ arrive , i want to rule out the CAN cables to be main issue

infiniteloop

@developeralgo222

i want to rule out the CAN cables to be main issue

Don’t put too much hope in the new cables. When @droftarts published the schematics of the CAN interface above, I understood that @DC42 implemented the most sophisticated circuity I can think of - it provides a carefully decoupled, true differential signal that should work even if you replace the dedicated copper wiring with wet clotheslines

Well, not quite so, but maybe we have to look for another root cause of the problem you experience with CAN.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

i want to rule out the CAN cables to be main issue

Don’t put too much hope in the new cables. When @droftarts published the schematics of the CAN interface above, I understood that @DC42 implemented the most sophisticated circuity I can think of - it provides a carefully decoupled, true differential signal that should work even if you replace the dedicated copper wiring with wet clotheslines

Well, not quite so, but maybe we have to look for another root cause of the problem you experience with CAN.

So i decided to play around with the CAN cables and made similar cables to Kenable ( as per information on their website which are CAT 5 / 5e , 4-pairs UTP, 26 AWG stranded , 100 Mhz cables. Changed all cables to the new similar cables

Now they don't Sync at all. Can @dc42 & @droftarts , chime in and let us know what might be going on. You guys are the designers after all. Leads me to believe its not the CAN cables, it might not even be the PWR issue we are thinking about . So what is it ? Has Duet 3 3HC Version 1.02 extensively been tested for HW design bugs or flaws ?

1/27/2024, 4:03:33 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:33 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:33 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:32 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:32 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:29 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:29 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:27 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:27 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:24 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:24 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:21 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:19 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:19 PM	Connected to 10.0.0.200

i also wanted to go through the CAN Bus troubleshooting so i measured the resistance at different points (stubs) on the CAN Bus . But can @dc42 or @droftarts , shed some light on what might be going on with CAN Bus here .