Duet 3HC Expansion looses Connection

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Twisted pair cables terminated in RJ11 connectors are sold in some countries as "High Speed ADSL cables". One supplier of such cables is Kenable. Note, ADSL and telephone cables made with flat wire often cross the connections, making them unsuitable

The price of kenable cables in US is so expensive while in UK they are extremely cheap. Can anyone point me to a link on Amazon or ebay for cable that works with Duet 3HC or 6XD CAN Bus without me cutting and remaking it, and is affordable

droftarts

@developeralgo222 I'm sorry you're having so many problems with this. It really shouldn't be this difficult, and it has us foxed as well! Maybe the best option is to return to the last known working configuration, which was as a bench setup.

For cables in the USA on Amazon, these look like the correct ones: https://www.amazon.com/Kenable-Speed-Broadband-Modem-Cable/dp/B011SS7LYU

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I'm sorry you're having so many problems with this. It really shouldn't be this difficult, and it has us foxed as well! Maybe the best option is to return to the last known working configuration, which was as a bench setup.

For cables in the USA on Amazon, these look like the correct ones: https://www.amazon.com/Kenable-Speed-Broadband-Modem-Cable/dp/B011SS7LYU

Ian

I agree this should not be this difficult. I have checked and double checked a lot of things but i can't get the CAN Sync to work. I decided to close my eyes, curse as much as i can and then ordered those cables let's hope that those help. Will get them in a week.

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I'm sorry you're having so many problems with this. It really shouldn't be this difficult, and it has us foxed as well! Maybe the best option is to return to the last known working configuration, which was as a bench setup.

For cables in the USA on Amazon, these look like the correct ones: https://www.amazon.com/Kenable-Speed-Broadband-Modem-Cable/dp/B011SS7LYU

Ian

I agree this should not be this difficult. I have checked and double checked a lot of things but i can't get the CAN Sync to work. I decided to close my eyes, curse as much as i can and then ordered those cables let's hope that those help. Will get them in a week.

@dc42, @droftarts , What is the expected terminating resistors values at the end of the CAN Bus for Duet boards? Ideally , Each resistor should ideally be 120 ohm, for a 60 ohm difference between CANH and CANL. When i measure the resistance with Multimeter on the terminated 3HC Board i get a difference of 70.4 Ohms and for Unterminated CAN_H & CAN_L i get 126.6 Ohms ? are this the expected values ?

Was doing some research and to continue troubleshooting, came across some explanation
General CAN Bus wiring and troubleshooting concepts document

Other CAN Bus terminating resistor values can be used and calculated based on specific baudrates and bus lengths - but, as you increase the baudrate, the pickier it gets about correct termination, whereas on lower baudrates you can get away with missing terminating resistors, by chance

Looking at the Duet Board / Schematic - i see Shield_GND terminal on the main board 6XD which i assume is for Ethernet Grounding.

Where is the CAN Grounding on Duet Boards ? since The CAN_GND of the CAN network should be connected to the functional earth potential (FE) at only one point i.e either on Main Board ( 6XD ) or on the Last 3HC on the Chain . Looking at the Kicad PCB design for 6XD , i see CAN_SHD_GND is that the same as CAN_GND in Duet 6XD design ?

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

Is there anything I might be forgetting on this GND solution. I am trying to avoid creating GND loops

That’s complex. As a general rule, imagine a star topology, I.e. for each and every voltage, there should be a single point (or rail) from where to further route the power lines.

For your dual PSUs, you already have this central point: V+ (1 rail with 3 connectors on each PSU) is now interconnected as discussed above in this thread.

Now, if you connect GND (of the DC-output) to Earth, do it at a single point as well. Think of it as a common rail where all GND lines meet - kind of a ”local Earth”. Don’t take GND or V– from distant parts of the frame. Aside from the common rail, consider them as not being the same.

Voltage lines may be split or chained on their run, but may then never refer to another source of the same voltage again (that’s a loop). You often run into this trap with shielded lines: the shield carries GND (or, in some cases, V–), but if you supply GND from both ends, you might involuntarily establish a ground loop. The golden rule here is to always feed GND from the origin - leave it unconnected at the destination end.

The fact that most devices in your machine are controlled via V– complicates things. All switched V– lines from the DUETs may never have contact with the ”local Earth”, GND or any other V–.

Grounding the PnP machine is another topic: not all metal parts are always in touch with each other, there are potential isolators such as coatings, belts or lubricants in the way, some parts are clearly isolated by design (printed joints etc). Short cable bridges help in some cases, but for moving parts, you might need to route a dedicated GND line through your cable chain.

Considering USB ground loops, I recommend this article from the documentation: USB ground loops

@infiniteloop , @dc42 , @droftarts

Can you give detail on this ? Currently i have the 2 PSUs connected like this

PSUs_Negative_Output_Connection_With_DC_to_AC_GND_1.png

Do i need to connect anything differently here ?

infiniteloop

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Can you give detail on this ? Currently i have the 2 PSUs connected like this

I’d rather like to take the Monty Python turn: And now for something completely different …

Well, let’s see where we are:

your wiring of the PSUs looks OK.
the test setup for CAN with isolated boards works - am I right with this?
if mounted on the PnP machine, only the CAN connection 6XD <-> 3HC „1“ works, all other connections fail - am I right with this?

If this is the current state of affairs, I’d like you to run another test: As of the photo below, the boards are stacked with electrically conductive standoffs who are, in turn, attached to the frame of the PnP machine. Now, can you detach the whole stack from the machine - but keep the standoffs between the boards in place? I’d like to know whether CAN becomes dysfunctional due to the GND it gets from the PnP machine or due to the electrical interconnection of the boards via the standoffs alone.

Please be assured that me, like @droftarts, am sorry for the problems you face. More so, I’m sincerely foxed by my inability to spot the root cause of the mystic CAN bus failure.

droftarts

@infiniteloop The stack of boards in that image are standing on what looks like a plastic toolbox, not connected to the machine.

As far as I can tell, there is no copper in close proximity to the mounting holes, at least on the 3HC v1.0 I have in my hands. There's no exposed copper around the holes, no via through-plating, and it looks like the copper ground plane doesn't extend to the holes. So I'm pretty sure they are electrically isolated from each other.

@developeralgo222 Plugging the two power supplies in to the same power bar should be fine. While we don't advise the same thing to get around USB ground loops, putting the two earths as close to each other as possible should largely mitigate a ground loop, if the PSUs are similar. I don't know if there's something about the phasing between them causing a problem; I'm no electrical engineer!

However... to mitigate that, have you tried powering all boards from one PSU, at least for testing? One PSU should have plenty of current to power all the boards, so long as no demanding load is put on it. Boards usually draw around 200-300mA with no peripherals connected.

For the CAN question, I don't think termination resistors values need changing, because we (a) run at 1Mbit (ie not that fast) and (b) your CAN bus is short (less than five meters). The 70.4 Ohm reading on the terminated 3HC might be interesting; can you check the resistance in the same way on all the boards, and see if there has been damage to the R37 and R38? These are two tiny 59R resistors (I think they work in series to be around 120R) between the CAN RJ11 ports and the CAN termination jumpers. Highlighted red (from the ibom here) :

This is the circuit diagram:

Though it may just be a measurement error.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@infiniteloop The stack of boards in that image are standing on what looks like a plastic toolbox, not connected to the machine.

As far as I can tell, there is no copper in close proximity to the mounting holes, at least on the 3HC v1.0 I have in my hands. There's no exposed copper around the holes, no via through-plating, and it looks like the copper ground plane doesn't extend to the holes. So I'm pretty sure they are electrically isolated from each other.

@developeralgo222 Plugging the two power supplies in to the same power bar should be fine. While we don't advise the same thing to get around USB ground loops, putting the two earths as close to each other as possible should largely mitigate a ground loop, if the PSUs are similar. I don't know if there's something about the phasing between them causing a problem; I'm no electrical engineer!

However... to mitigate that, have you tried powering all boards from one PSU, at least for testing? One PSU should have plenty of current to power all the boards, so long as no demanding load is put on it. Boards usually draw around 200-300mA with no peripherals connected.

For the CAN question, I don't think termination resistors values need changing, because we (a) run at 1Mbit (ie not that fast) and (b) your CAN bus is short (less than five meters). The 70.4 Ohm reading on the terminated 3HC might be interesting; can you check the resistance in the same way on all the boards, and see if there has been damage to the R37 and R38? These are two tiny 59R resistors (I think they work in series to be around 120R) between the CAN RJ11 ports and the CAN termination jumpers. Highlighted red (from the ibom here) :

This is the circuit diagram:

Though it may just be a measurement error.

Ian

When i measure resistance across unterminated 3HC boards CAN_H & CAN_L i get 140 Ohms for Board 1, 2, 3 and then Board 4 is 70 Ohms which is correct because its terminated and that's the differential resistance. Early in my troubleshooting phase, i did try to power all from one PSU but that did not make a difference. i will try again for testing purpose .

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Can you give detail on this ? Currently i have the 2 PSUs connected like this

I’d rather like to take the Monty Python turn: And now for something completely different …

Well, let’s see where we are:

your wiring of the PSUs looks OK.

the test setup for CAN with isolated boards works - am I right with this?

if mounted on the PnP machine, only the CAN connection 6XD <-> 3HC „1“ works, all other connections fail - am I right with this?

If this is the current state of affairs, I’d like you to run another test: As of the photo below, the boards are stacked with electrically conductive standoffs who are, in turn, attached to the frame of the PnP machine. Now, can you detach the whole stack from the machine - but keep the standoffs between the boards in place? I’d like to know whether CAN becomes dysfunctional due to the GND it gets from the PnP machine or due to the electrical interconnection of the boards via the standoffs alone.

Please be assured that me, like @droftarts, am sorry for the problems you face. More so, I’m sincerely foxed by my inability to spot the root cause of the mystic CAN bus failure.

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards. They no longer touch the frame or any metal attached to the frame.

infiniteloop

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards.

I propose to use this setup (isolated stack, mounted on the PnP) for further tests:

follow @droftarts' proposal and use just a single PSU
with all peripherals detached, just VIN and CAN connected, are all boards recognised?
if not, use the short CAN cables from your previous test setup.

At least the results of the last test should match those from your previous test setup on the bench: all boards should connect. If not, there must be a difference we are not aware of. Let's see, please share the results.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards.

I propose to use this setup (isolated stack, mounted on the PnP) for further tests:

follow @droftarts' proposal and use just a single PSU

with all peripherals detached, just VIN and CAN connected, are all boards recognised?

if not, use the short CAN cables from your previous test setup.

At least the results of the last test should match those from your previous test setup on the bench: all boards should connect. If not, there must be a difference we are not aware of. Let's see, please share the results.

For testing & Troubleshooting

i have connected all the boards ( 6XD plus 4x 3HC ) to a single DC PSU ( 24V , 20A )
Only Power ( VIN & GND ) and CAN Bus are connected.
i used a short standard Phone cable between 6XD & 3HC1 (that i cut to a straight Cable since phone cable are crossed)

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Test results:

All boards sync lights are synchronized but i get the following errors

1/26/2024, 5:48:48 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:48:06 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:47:04 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:23:04 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:14:12 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:13:07 PM	Error: Expansion board 1 stopped sending status

i think there might be PSU issue plus CAN cable issue, Once i get the recommended Kenable ADSL 2+ cables i should be able to completely rule out CAN Cable issue.

Even with a ray of hope it still doesn't work as expected

infiniteloop

@developeralgo222

Error: Expansion board 1 stopped sending status

That's one for @droftarts or @dc42, I am unable to draw any qualified conclusions from that. Hm, what's special about board #1? Is this by chance the same board which you, on January 19, reported as being the only one that synchronises with the 6XD?

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Yes, you mentioned earlier the delivery could take a week or so. Maybe that with these, the results of the initial test setup can be reproduced - prove me wrong, but I think the greater distance between the 6XD and the stack now is the only difference to the initial bench test.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

Error: Expansion board 1 stopped sending status

That's one for @droftarts or @dc42, I am unable to draw any qualified conclusions from that. Hm, what's special about board #1? Is this by chance the same board which you, on January 19, reported as being the only one that synchronises with the 6XD?

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Yes, you mentioned earlier the delivery could take a week or so. Maybe that with these, the results of the initial test setup can be reproduced - prove me wrong, but I think the greater distance between the 6XD and the stack now is the only difference to the initial bench test.

Yes it's the same 3HC board. I think the Cable from 6XD to 3HC1 might be having issues too. Once the new Kenable ADSL 2+ arrive , i want to rule out the CAN cables to be main issue

infiniteloop

@developeralgo222

i want to rule out the CAN cables to be main issue

Don’t put too much hope in the new cables. When @droftarts published the schematics of the CAN interface above, I understood that @DC42 implemented the most sophisticated circuity I can think of - it provides a carefully decoupled, true differential signal that should work even if you replace the dedicated copper wiring with wet clotheslines

Well, not quite so, but maybe we have to look for another root cause of the problem you experience with CAN.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

i want to rule out the CAN cables to be main issue

Don’t put too much hope in the new cables. When @droftarts published the schematics of the CAN interface above, I understood that @DC42 implemented the most sophisticated circuity I can think of - it provides a carefully decoupled, true differential signal that should work even if you replace the dedicated copper wiring with wet clotheslines

Well, not quite so, but maybe we have to look for another root cause of the problem you experience with CAN.

So i decided to play around with the CAN cables and made similar cables to Kenable ( as per information on their website which are CAT 5 / 5e , 4-pairs UTP, 26 AWG stranded , 100 Mhz cables. Changed all cables to the new similar cables

Now they don't Sync at all. Can @dc42 & @droftarts , chime in and let us know what might be going on. You guys are the designers after all. Leads me to believe its not the CAN cables, it might not even be the PWR issue we are thinking about . So what is it ? Has Duet 3 3HC Version 1.02 extensively been tested for HW design bugs or flaws ?

1/27/2024, 4:03:33 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:33 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:33 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:32 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:32 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:29 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:29 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:27 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:27 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:24 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:24 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:21 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:19 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:19 PM	Connected to 10.0.0.200

i also wanted to go through the CAN Bus troubleshooting so i measured the resistance at different points (stubs) on the CAN Bus . But can @dc42 or @droftarts , shed some light on what might be going on with CAN Bus here .

infiniteloop

@developeralgo222

So what is it?

Good question. Currently, we are just faced with some boards who will not always sync. @droftarts detected the missing GND link with the PSUs, @DC42 spotted the CAN cables as a possible point of failure. Let’s wait for the arrival of the Kenable cables to verify (or falsify) his thesis.

In the meantime, you came up with three configurations:

all boards on a bench (test setup)
all boards integrated with the PnP machine (including the machine’s CAN cabling)
all boards mounted in place (isolated stack), but with just CAN and VIN cabling attached

We got mixed results, just config #1 works reliably (with respect to CAN). In all other aspects, this mix of configurations represents a moving target, with a lot of details changing along the way. This makes it difficult to focus on a single trace at a time: too many variables who might affect the results but go unnoticed. Admittedly, I contributed to the confusion.

At the moment, it looks like the setup is extremely sensible to which specific CAN cables are used. That’s quite unexpected, but let’s wait for the known-good cables to arrive.

Addendum:

Has Duet 3 3HC Version 1.02 extensively been tested for HW design bugs or flaws ?

Sure. BTW, you approved proper functionality of the boards with the aforementioned test setup (#1).

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

So what is it?

Good question. Currently, we are just faced with some boards who will not always sync. @droftarts detected the missing GND link with the PSUs, @DC42 spotted the CAN cables as a possible point of failure. Let’s wait for the arrival of the Kenable cables to verify (or falsify) his thesis.

In the meantime, you came up with three configurations:

all boards on a bench (test setup)

all boards integrated with the PnP machine (including the machine’s CAN cabling)

all boards mounted in place (isolated stack), but with just CAN and VIN cabling attached

We got mixed results, just config #1 works reliably (with respect to CAN). In all other aspects, this mix of configurations represents a moving target, with a lot of details changing along the way. This makes it difficult to focus on a single trace at a time: too many variables who might affect the results but go unnoticed. Admittedly, I contributed to the confusion.

At the moment, it looks like the setup is extremely sensible to which specific CAN cables are used. That’s quite unexpected, but let’s wait for the known-good cables to arrive.

Addendum:

Has Duet 3 3HC Version 1.02 extensively been tested for HW design bugs or flaws ?

Sure. BTW, you approved proper functionality of the boards with the aforementioned test setup (#1).

Just to clarify : There is no moving target and i know its frustrating for both me and the forum support. It has been over 2 weeks of simply trying to figure out why CAN Bus connection for Duet 3 6XD and 4 x 3HC is not synchronizing / Not working as expected. All the tests have been to verify and we don't just want to make assumptions but to rule out the most common issues during this tough and frustrating troubleshooting phase.

The Probable cause might be :

(1) CAN Bus Issue
(2) PSU Issue ( 1 PSU vs 2 PSUs )
(3) Grounding issue including CAN_GND
(4) Anything else ??

All the tests below are meant to Rule out most of the common issues above that might be causing the CAN Bus synchronization of the boards not to work.

All boards on a bench (test setup)
All boards integrated with the PnP machine (including the machine’s CAN cabling & VIN ) - This and next below are same test just isolated the stack to avoid any grounding issues if available. Nothing is connected to the boards apart from CAN & VIN cabling attched
All boards mounted in place (isolated stack), but with just CAN and VIN cabling attached

As i said before , i hope the Kenable ADSL 2+ cable can rule out any CAN Bus Cable issues. I don't think so but its always better to rule it out .

infiniteloop

@developeralgo222

i know its frustrating for both me and the forum support.

True, frustration is the dominant feeling when I’m hunting for bugs in my hard- or software. Things become more complicated if large parts of the system are a terra incognita, i.e. unknown to me. Without better knowledge of your wiring (how you integrate the Duets into the machine), I can assume your setup to be "right" (which, in itself, may be debatable), or make other assumptions, but I can’t do without these unless I know and comprehend all potentially relevant facts. Sorry for the inconvenience, but I know of no other way to deal with errors that simply shouldn’t be there.

RE 2 or 3 cases:

This and next below are same test

In the context of the isolated stack, you posted a photo (see below, Jan 26th) with several sockets populated. That’s different from ”just CAN and VIN”. Whether this is relevant or not… well, that depends on my assumptions about your electrical layout. You are the only one who knows the PnP in detail.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

i know its frustrating for both me and the forum support.

True, frustration is the dominant feeling when I’m hunting for bugs in my hard- or software. Things become more complicated if large parts of the system are a terra incognita, i.e. unknown to me. Without better knowledge of your wiring (how you integrate the Duets into the machine), I can assume your setup to be "right" (which, in itself, may be debatable), or make other assumptions, but I can’t do without these unless I know and comprehend all potentially relevant facts. Sorry for the inconvenience, but I know of no other way to deal with errors that simply shouldn’t be there.

RE 2 or 3 cases:

This and next below are same test

In the context of the isolated stack, you posted a photo (see below, Jan 26th) with several sockets populated. That’s different from ”just CAN and VIN”. Whether this is relevant or not… well, that depends on my assumptions about your electrical layout. You are the only one who knows the PnP in detail.

Sorry for the confusion on the photos. Yes the photo was to show the isolation of the stack. It's not Powered at all in that photo. When i am testing nothing is connected apart from the VIN & CAN Bus cables. Its frustrating and i wish i could find the problem causing CAN Bus sync issue and not have to ask for help from the designers and support. i have tried every suggested solution or test and CAN Bus Synchronization has not worked as expected

NOTE: Just received the first batch of Cables today and i replaced the 6XD --> 3HC1 cable with 15m Kenable ADSL 2+ and that did not make any difference . No sync at all . As i thought before i don't think it's the Cables themselves. i am also still waiting for 3 short Kenable Cables to replace the CAN Bus connections between the 3HC boards. But i don't think that will change anything at the moment.

Decided to retest the isolated boards (Test bench setup ) by removing them from the PnP machine using kenable ADSL 2+ cable from 6XD ---> 3HC1 . It does not work. Stacked or Not does not matter it does not work. Tested individual 3HC boards they also do not sync with 6XD using Kenable ADSL 2+ cable

Tested Long 15m Kenable ADSL 2+ between 6XD --> 3HC1 ---- The test bench setup does not work
Tested self-made short Cable 0.5m (50cm) between 6XD --> 3HC1 ---- The test bench setup works and syncs without any errors so far
Tested self-made short Cable 4m (50cm) between 6XD --> 3HC1 ---- The test bench setup works and syncs without any errors so far

Now trying to test different lengths between 6XD --> 3HC1 on the isolated test bench . Frankly i am not sure what to make of this results . CAN Bus says a length upto 40m @ 1 Mb/s rate is fine

1/28/2024, 2:25:16 PM	

M122 B0

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode)
Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 153284
Dynamic ram: 117992 of which 0 recycled
Never used RAM 71860, free system stack 202 words
Tasks: NETWORK(1,ready,31.1%,182) ETHERNET(5,nWait,0.0%,321) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,351) MAIN(1,running,68.8%,444) IDLE(0,ready,0.0%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:11:10 ago, cause: power up
Last software reset at 2024-01-19 15:33, reason: User, Gcodes spinning, available RAM 71300, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 28.0, current 43.3, max 43.4
Supply voltage: min 24.0, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 2024-01-28 14:25:15
Slowest loop: 24.13ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 3455, received 21664, lost 0, errs 217, boc 0
Longest wait 2ms for reply type 6024, peak Tx sync delay 252, free buffers 50 (min 49), ts 3353/3352/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 7.14ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0


1/28/2024, 2:25:12 PM	

M122 B4

Diagnostics for board 4:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156952, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,399) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:11:06 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43296, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43297, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43297, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/7, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.3, current 24.3, max 24.4
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 22.9C, current 30.4C, max 30.6C
Last sensors broadcast 0x00000000 found 0 38 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 5357, send timeouts 0, received 3342, lost 0, errs 223, boc 0, free buffers 38, min 38, error reg ff0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 2:25:09 PM	

M122 B3

Diagnostics for board 3:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156832, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:11:03 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32406, writes 17 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32407, writes 17 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32408, writes 17 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -5/7, peak Rx sync delay 183, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 21.3, current 24.2, max 24.3
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 22.8C, current 30.0C, max 30.0C
Last sensors broadcast 0x00000000 found 0 242 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 5350, send timeouts 0, received 3345, lost 0, errs 223, boc 0, free buffers 38, min 38, error reg ff0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 2:25:06 PM	

M122 B2

Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156832, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,421) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:11:01 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21924, writes 17 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21924, writes 17 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21925, writes 17 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/6, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.3
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 22.8C, current 31.6C, max 31.6C
Last sensors broadcast 0x00000000 found 0 66 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 5330, send timeouts 0, received 3331, lost 0, errs 217, boc 0, free buffers 38, min 38, error reg ff0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 2:25:03 PM	

M122 B1

Diagnostics for board 1:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156904, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:10:57 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59337, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59336, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59336, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -3/5, peak Rx sync delay 177, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.1, current 24.1, max 24.1
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 19.6C, current 26.8C, max 26.8C
Last sensors broadcast 0x00000000 found 0 83 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 152, send timeouts 0, received 84, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

infiniteloop

@developeralgo222

Decided to retest the isolated boards (Test bench setup )

Thank you for the tests. Looks like some meters of CAN more or less make a difference - that’s not OK at all. Could you please run two quick tests on the bench, both starting with the setup as on the photo:

physically swap 3HC boards #1 and #2 in the CAN chain, take the 15m cable to link the 6XD with 3HC2 (chaining 6XD -> 3HC2 -> 3HC1 -> 3HC3 -> 3HC4). No change of IDs. This is to see whether a different CAN port on at least one end of the connection with the 6XD makes a difference or not.
again starting with the initial arrangement, replace the short cable between 3HC3 and 3HC4 with the 15m one. The result might indicate a problem with either the overall length of the CAN bus, or the position of the long section within the chain of boards.

For both tests, would you mind to post the M122 reports (if available)?