Duet 3HC Expansion looses Connection

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

Is there anything I might be forgetting on this GND solution. I am trying to avoid creating GND loops

That’s complex. As a general rule, imagine a star topology, I.e. for each and every voltage, there should be a single point (or rail) from where to further route the power lines.

For your dual PSUs, you already have this central point: V+ (1 rail with 3 connectors on each PSU) is now interconnected as discussed above in this thread.

Now, if you connect GND (of the DC-output) to Earth, do it at a single point as well. Think of it as a common rail where all GND lines meet - kind of a ”local Earth”. Don’t take GND or V– from distant parts of the frame. Aside from the common rail, consider them as not being the same.

Voltage lines may be split or chained on their run, but may then never refer to another source of the same voltage again (that’s a loop). You often run into this trap with shielded lines: the shield carries GND (or, in some cases, V–), but if you supply GND from both ends, you might involuntarily establish a ground loop. The golden rule here is to always feed GND from the origin - leave it unconnected at the destination end.

The fact that most devices in your machine are controlled via V– complicates things. All switched V– lines from the DUETs may never have contact with the ”local Earth”, GND or any other V–.

Grounding the PnP machine is another topic: not all metal parts are always in touch with each other, there are potential isolators such as coatings, belts or lubricants in the way, some parts are clearly isolated by design (printed joints etc). Short cable bridges help in some cases, but for moving parts, you might need to route a dedicated GND line through your cable chain.

Considering USB ground loops, I recommend this article from the documentation: USB ground loops

@infiniteloop , @dc42 , @droftarts

Can you give detail on this ? Currently i have the 2 PSUs connected like this

PSUs_Negative_Output_Connection_With_DC_to_AC_GND_1.png

Do i need to connect anything differently here ?

infiniteloop

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Can you give detail on this ? Currently i have the 2 PSUs connected like this

I’d rather like to take the Monty Python turn: And now for something completely different …

Well, let’s see where we are:

your wiring of the PSUs looks OK.
the test setup for CAN with isolated boards works - am I right with this?
if mounted on the PnP machine, only the CAN connection 6XD <-> 3HC „1“ works, all other connections fail - am I right with this?

If this is the current state of affairs, I’d like you to run another test: As of the photo below, the boards are stacked with electrically conductive standoffs who are, in turn, attached to the frame of the PnP machine. Now, can you detach the whole stack from the machine - but keep the standoffs between the boards in place? I’d like to know whether CAN becomes dysfunctional due to the GND it gets from the PnP machine or due to the electrical interconnection of the boards via the standoffs alone.

Please be assured that me, like @droftarts, am sorry for the problems you face. More so, I’m sincerely foxed by my inability to spot the root cause of the mystic CAN bus failure.

droftarts

@infiniteloop The stack of boards in that image are standing on what looks like a plastic toolbox, not connected to the machine.

As far as I can tell, there is no copper in close proximity to the mounting holes, at least on the 3HC v1.0 I have in my hands. There's no exposed copper around the holes, no via through-plating, and it looks like the copper ground plane doesn't extend to the holes. So I'm pretty sure they are electrically isolated from each other.

@developeralgo222 Plugging the two power supplies in to the same power bar should be fine. While we don't advise the same thing to get around USB ground loops, putting the two earths as close to each other as possible should largely mitigate a ground loop, if the PSUs are similar. I don't know if there's something about the phasing between them causing a problem; I'm no electrical engineer!

However... to mitigate that, have you tried powering all boards from one PSU, at least for testing? One PSU should have plenty of current to power all the boards, so long as no demanding load is put on it. Boards usually draw around 200-300mA with no peripherals connected.

For the CAN question, I don't think termination resistors values need changing, because we (a) run at 1Mbit (ie not that fast) and (b) your CAN bus is short (less than five meters). The 70.4 Ohm reading on the terminated 3HC might be interesting; can you check the resistance in the same way on all the boards, and see if there has been damage to the R37 and R38? These are two tiny 59R resistors (I think they work in series to be around 120R) between the CAN RJ11 ports and the CAN termination jumpers. Highlighted red (from the ibom here) :

This is the circuit diagram:

Though it may just be a measurement error.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@infiniteloop The stack of boards in that image are standing on what looks like a plastic toolbox, not connected to the machine.

As far as I can tell, there is no copper in close proximity to the mounting holes, at least on the 3HC v1.0 I have in my hands. There's no exposed copper around the holes, no via through-plating, and it looks like the copper ground plane doesn't extend to the holes. So I'm pretty sure they are electrically isolated from each other.

@developeralgo222 Plugging the two power supplies in to the same power bar should be fine. While we don't advise the same thing to get around USB ground loops, putting the two earths as close to each other as possible should largely mitigate a ground loop, if the PSUs are similar. I don't know if there's something about the phasing between them causing a problem; I'm no electrical engineer!

However... to mitigate that, have you tried powering all boards from one PSU, at least for testing? One PSU should have plenty of current to power all the boards, so long as no demanding load is put on it. Boards usually draw around 200-300mA with no peripherals connected.

For the CAN question, I don't think termination resistors values need changing, because we (a) run at 1Mbit (ie not that fast) and (b) your CAN bus is short (less than five meters). The 70.4 Ohm reading on the terminated 3HC might be interesting; can you check the resistance in the same way on all the boards, and see if there has been damage to the R37 and R38? These are two tiny 59R resistors (I think they work in series to be around 120R) between the CAN RJ11 ports and the CAN termination jumpers. Highlighted red (from the ibom here) :

This is the circuit diagram:

Though it may just be a measurement error.

Ian

When i measure resistance across unterminated 3HC boards CAN_H & CAN_L i get 140 Ohms for Board 1, 2, 3 and then Board 4 is 70 Ohms which is correct because its terminated and that's the differential resistance. Early in my troubleshooting phase, i did try to power all from one PSU but that did not make a difference. i will try again for testing purpose .

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

Can you give detail on this ? Currently i have the 2 PSUs connected like this

I’d rather like to take the Monty Python turn: And now for something completely different …

Well, let’s see where we are:

your wiring of the PSUs looks OK.

the test setup for CAN with isolated boards works - am I right with this?

if mounted on the PnP machine, only the CAN connection 6XD <-> 3HC „1“ works, all other connections fail - am I right with this?

If this is the current state of affairs, I’d like you to run another test: As of the photo below, the boards are stacked with electrically conductive standoffs who are, in turn, attached to the frame of the PnP machine. Now, can you detach the whole stack from the machine - but keep the standoffs between the boards in place? I’d like to know whether CAN becomes dysfunctional due to the GND it gets from the PnP machine or due to the electrical interconnection of the boards via the standoffs alone.

Please be assured that me, like @droftarts, am sorry for the problems you face. More so, I’m sincerely foxed by my inability to spot the root cause of the mystic CAN bus failure.

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards. They no longer touch the frame or any metal attached to the frame.

infiniteloop

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards.

I propose to use this setup (isolated stack, mounted on the PnP) for further tests:

follow @droftarts' proposal and use just a single PSU
with all peripherals detached, just VIN and CAN connected, are all boards recognised?
if not, use the short CAN cables from your previous test setup.

At least the results of the last test should match those from your previous test setup on the bench: all boards should connect. If not, there must be a difference we are not aware of. Let's see, please share the results.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

i have updated the stands to Nylon standoff and completely isolated them using Nylon standoffs in between the boards.

I propose to use this setup (isolated stack, mounted on the PnP) for further tests:

follow @droftarts' proposal and use just a single PSU

with all peripherals detached, just VIN and CAN connected, are all boards recognised?

if not, use the short CAN cables from your previous test setup.

At least the results of the last test should match those from your previous test setup on the bench: all boards should connect. If not, there must be a difference we are not aware of. Let's see, please share the results.

For testing & Troubleshooting

i have connected all the boards ( 6XD plus 4x 3HC ) to a single DC PSU ( 24V , 20A )
Only Power ( VIN & GND ) and CAN Bus are connected.
i used a short standard Phone cable between 6XD & 3HC1 (that i cut to a straight Cable since phone cable are crossed)

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Test results:

All boards sync lights are synchronized but i get the following errors

1/26/2024, 5:48:48 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:48:06 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:47:04 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:23:04 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:14:12 PM	Error: Expansion board 1 stopped sending status
1/26/2024, 5:13:07 PM	Error: Expansion board 1 stopped sending status

i think there might be PSU issue plus CAN cable issue, Once i get the recommended Kenable ADSL 2+ cables i should be able to completely rule out CAN Cable issue.

Even with a ray of hope it still doesn't work as expected

infiniteloop

@developeralgo222

Error: Expansion board 1 stopped sending status

That's one for @droftarts or @dc42, I am unable to draw any qualified conclusions from that. Hm, what's special about board #1? Is this by chance the same board which you, on January 19, reported as being the only one that synchronises with the 6XD?

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Yes, you mentioned earlier the delivery could take a week or so. Maybe that with these, the results of the initial test setup can be reproduced - prove me wrong, but I think the greater distance between the 6XD and the stack now is the only difference to the initial bench test.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

Error: Expansion board 1 stopped sending status

That's one for @droftarts or @dc42, I am unable to draw any qualified conclusions from that. Hm, what's special about board #1? Is this by chance the same board which you, on January 19, reported as being the only one that synchronises with the 6XD?

Note: New recommended Kenable ADSL 2+ Cables that i ordered have not arrived yet until Sunday or Monday.

Yes, you mentioned earlier the delivery could take a week or so. Maybe that with these, the results of the initial test setup can be reproduced - prove me wrong, but I think the greater distance between the 6XD and the stack now is the only difference to the initial bench test.

Yes it's the same 3HC board. I think the Cable from 6XD to 3HC1 might be having issues too. Once the new Kenable ADSL 2+ arrive , i want to rule out the CAN cables to be main issue

infiniteloop

@developeralgo222

i want to rule out the CAN cables to be main issue

Don’t put too much hope in the new cables. When @droftarts published the schematics of the CAN interface above, I understood that @DC42 implemented the most sophisticated circuity I can think of - it provides a carefully decoupled, true differential signal that should work even if you replace the dedicated copper wiring with wet clotheslines

Well, not quite so, but maybe we have to look for another root cause of the problem you experience with CAN.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

i want to rule out the CAN cables to be main issue

Don’t put too much hope in the new cables. When @droftarts published the schematics of the CAN interface above, I understood that @DC42 implemented the most sophisticated circuity I can think of - it provides a carefully decoupled, true differential signal that should work even if you replace the dedicated copper wiring with wet clotheslines

Well, not quite so, but maybe we have to look for another root cause of the problem you experience with CAN.

So i decided to play around with the CAN cables and made similar cables to Kenable ( as per information on their website which are CAT 5 / 5e , 4-pairs UTP, 26 AWG stranded , 100 Mhz cables. Changed all cables to the new similar cables

Now they don't Sync at all. Can @dc42 & @droftarts , chime in and let us know what might be going on. You guys are the designers after all. Leads me to believe its not the CAN cables, it might not even be the PWR issue we are thinking about . So what is it ? Has Duet 3 3HC Version 1.02 extensively been tested for HW design bugs or flaws ?

1/27/2024, 4:03:33 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:33 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:33 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:32 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:32 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:31 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:30 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:29 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:29 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:28 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:27 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:27 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
1/27/2024, 4:03:26 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:25 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:24 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:24 PM	Error: Expansion board 3 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:23 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:22 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:21 PM	Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
			Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
			Error: Expansion board 2 reconnected
			Error: Expansion board 3 reconnected
1/27/2024, 4:03:19 PM	Error: Expansion board 4 reconnected
			Error: Expansion board 1 reconnected
1/27/2024, 4:03:19 PM	Connected to 10.0.0.200

i also wanted to go through the CAN Bus troubleshooting so i measured the resistance at different points (stubs) on the CAN Bus . But can @dc42 or @droftarts , shed some light on what might be going on with CAN Bus here .

infiniteloop

@developeralgo222

So what is it?

Good question. Currently, we are just faced with some boards who will not always sync. @droftarts detected the missing GND link with the PSUs, @DC42 spotted the CAN cables as a possible point of failure. Let’s wait for the arrival of the Kenable cables to verify (or falsify) his thesis.

In the meantime, you came up with three configurations:

all boards on a bench (test setup)
all boards integrated with the PnP machine (including the machine’s CAN cabling)
all boards mounted in place (isolated stack), but with just CAN and VIN cabling attached

We got mixed results, just config #1 works reliably (with respect to CAN). In all other aspects, this mix of configurations represents a moving target, with a lot of details changing along the way. This makes it difficult to focus on a single trace at a time: too many variables who might affect the results but go unnoticed. Admittedly, I contributed to the confusion.

At the moment, it looks like the setup is extremely sensible to which specific CAN cables are used. That’s quite unexpected, but let’s wait for the known-good cables to arrive.

Addendum:

Has Duet 3 3HC Version 1.02 extensively been tested for HW design bugs or flaws ?

Sure. BTW, you approved proper functionality of the boards with the aforementioned test setup (#1).

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

So what is it?

Good question. Currently, we are just faced with some boards who will not always sync. @droftarts detected the missing GND link with the PSUs, @DC42 spotted the CAN cables as a possible point of failure. Let’s wait for the arrival of the Kenable cables to verify (or falsify) his thesis.

In the meantime, you came up with three configurations:

all boards on a bench (test setup)

all boards integrated with the PnP machine (including the machine’s CAN cabling)

all boards mounted in place (isolated stack), but with just CAN and VIN cabling attached

We got mixed results, just config #1 works reliably (with respect to CAN). In all other aspects, this mix of configurations represents a moving target, with a lot of details changing along the way. This makes it difficult to focus on a single trace at a time: too many variables who might affect the results but go unnoticed. Admittedly, I contributed to the confusion.

At the moment, it looks like the setup is extremely sensible to which specific CAN cables are used. That’s quite unexpected, but let’s wait for the known-good cables to arrive.

Addendum:

Has Duet 3 3HC Version 1.02 extensively been tested for HW design bugs or flaws ?

Sure. BTW, you approved proper functionality of the boards with the aforementioned test setup (#1).

Just to clarify : There is no moving target and i know its frustrating for both me and the forum support. It has been over 2 weeks of simply trying to figure out why CAN Bus connection for Duet 3 6XD and 4 x 3HC is not synchronizing / Not working as expected. All the tests have been to verify and we don't just want to make assumptions but to rule out the most common issues during this tough and frustrating troubleshooting phase.

The Probable cause might be :

(1) CAN Bus Issue
(2) PSU Issue ( 1 PSU vs 2 PSUs )
(3) Grounding issue including CAN_GND
(4) Anything else ??

All the tests below are meant to Rule out most of the common issues above that might be causing the CAN Bus synchronization of the boards not to work.

All boards on a bench (test setup)
All boards integrated with the PnP machine (including the machine’s CAN cabling & VIN ) - This and next below are same test just isolated the stack to avoid any grounding issues if available. Nothing is connected to the boards apart from CAN & VIN cabling attched
All boards mounted in place (isolated stack), but with just CAN and VIN cabling attached

As i said before , i hope the Kenable ADSL 2+ cable can rule out any CAN Bus Cable issues. I don't think so but its always better to rule it out .

infiniteloop

@developeralgo222

i know its frustrating for both me and the forum support.

True, frustration is the dominant feeling when I’m hunting for bugs in my hard- or software. Things become more complicated if large parts of the system are a terra incognita, i.e. unknown to me. Without better knowledge of your wiring (how you integrate the Duets into the machine), I can assume your setup to be "right" (which, in itself, may be debatable), or make other assumptions, but I can’t do without these unless I know and comprehend all potentially relevant facts. Sorry for the inconvenience, but I know of no other way to deal with errors that simply shouldn’t be there.

RE 2 or 3 cases:

This and next below are same test

In the context of the isolated stack, you posted a photo (see below, Jan 26th) with several sockets populated. That’s different from ”just CAN and VIN”. Whether this is relevant or not… well, that depends on my assumptions about your electrical layout. You are the only one who knows the PnP in detail.

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

i know its frustrating for both me and the forum support.

True, frustration is the dominant feeling when I’m hunting for bugs in my hard- or software. Things become more complicated if large parts of the system are a terra incognita, i.e. unknown to me. Without better knowledge of your wiring (how you integrate the Duets into the machine), I can assume your setup to be "right" (which, in itself, may be debatable), or make other assumptions, but I can’t do without these unless I know and comprehend all potentially relevant facts. Sorry for the inconvenience, but I know of no other way to deal with errors that simply shouldn’t be there.

RE 2 or 3 cases:

This and next below are same test

In the context of the isolated stack, you posted a photo (see below, Jan 26th) with several sockets populated. That’s different from ”just CAN and VIN”. Whether this is relevant or not… well, that depends on my assumptions about your electrical layout. You are the only one who knows the PnP in detail.

Sorry for the confusion on the photos. Yes the photo was to show the isolation of the stack. It's not Powered at all in that photo. When i am testing nothing is connected apart from the VIN & CAN Bus cables. Its frustrating and i wish i could find the problem causing CAN Bus sync issue and not have to ask for help from the designers and support. i have tried every suggested solution or test and CAN Bus Synchronization has not worked as expected

NOTE: Just received the first batch of Cables today and i replaced the 6XD --> 3HC1 cable with 15m Kenable ADSL 2+ and that did not make any difference . No sync at all . As i thought before i don't think it's the Cables themselves. i am also still waiting for 3 short Kenable Cables to replace the CAN Bus connections between the 3HC boards. But i don't think that will change anything at the moment.

Decided to retest the isolated boards (Test bench setup ) by removing them from the PnP machine using kenable ADSL 2+ cable from 6XD ---> 3HC1 . It does not work. Stacked or Not does not matter it does not work. Tested individual 3HC boards they also do not sync with 6XD using Kenable ADSL 2+ cable

Tested Long 15m Kenable ADSL 2+ between 6XD --> 3HC1 ---- The test bench setup does not work
Tested self-made short Cable 0.5m (50cm) between 6XD --> 3HC1 ---- The test bench setup works and syncs without any errors so far
Tested self-made short Cable 4m (50cm) between 6XD --> 3HC1 ---- The test bench setup works and syncs without any errors so far

Now trying to test different lengths between 6XD --> 3HC1 on the isolated test bench . Frankly i am not sure what to make of this results . CAN Bus says a length upto 40m @ 1 Mb/s rate is fine

1/28/2024, 2:25:16 PM	

M122 B0

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode)
Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 153284
Dynamic ram: 117992 of which 0 recycled
Never used RAM 71860, free system stack 202 words
Tasks: NETWORK(1,ready,31.1%,182) ETHERNET(5,nWait,0.0%,321) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,351) MAIN(1,running,68.8%,444) IDLE(0,ready,0.0%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:11:10 ago, cause: power up
Last software reset at 2024-01-19 15:33, reason: User, Gcodes spinning, available RAM 71300, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 28.0, current 43.3, max 43.4
Supply voltage: min 24.0, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 2024-01-28 14:25:15
Slowest loop: 24.13ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 3455, received 21664, lost 0, errs 217, boc 0
Longest wait 2ms for reply type 6024, peak Tx sync delay 252, free buffers 50 (min 49), ts 3353/3352/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 7.14ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0


1/28/2024, 2:25:12 PM	

M122 B4

Diagnostics for board 4:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156952, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,399) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:11:06 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43296, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43297, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43297, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/7, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.3, current 24.3, max 24.4
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 22.9C, current 30.4C, max 30.6C
Last sensors broadcast 0x00000000 found 0 38 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 5357, send timeouts 0, received 3342, lost 0, errs 223, boc 0, free buffers 38, min 38, error reg ff0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 2:25:09 PM	

M122 B3

Diagnostics for board 3:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156832, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:11:03 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32406, writes 17 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32407, writes 17 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32408, writes 17 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -5/7, peak Rx sync delay 183, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 21.3, current 24.2, max 24.3
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 22.8C, current 30.0C, max 30.0C
Last sensors broadcast 0x00000000 found 0 242 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 5350, send timeouts 0, received 3345, lost 0, errs 223, boc 0, free buffers 38, min 38, error reg ff0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 2:25:06 PM	

M122 B2

Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156832, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,421) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:11:01 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21924, writes 17 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21924, writes 17 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21925, writes 17 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -6/6, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.3
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 22.8C, current 31.6C, max 31.6C
Last sensors broadcast 0x00000000 found 0 66 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 5330, send timeouts 0, received 3331, lost 0, errs 217, boc 0, free buffers 38, min 38, error reg ff0000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 2:25:03 PM	

M122 B1

Diagnostics for board 1:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156904, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:10:57 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59337, writes 0 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59336, writes 0 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59336, writes 0 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -3/5, peak Rx sync delay 177, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.1, current 24.1, max 24.1
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 19.6C, current 26.8C, max 26.8C
Last sensors broadcast 0x00000000 found 0 83 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 152, send timeouts 0, received 84, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

infiniteloop

@developeralgo222

Decided to retest the isolated boards (Test bench setup )

Thank you for the tests. Looks like some meters of CAN more or less make a difference - that’s not OK at all. Could you please run two quick tests on the bench, both starting with the setup as on the photo:

physically swap 3HC boards #1 and #2 in the CAN chain, take the 15m cable to link the 6XD with 3HC2 (chaining 6XD -> 3HC2 -> 3HC1 -> 3HC3 -> 3HC4). No change of IDs. This is to see whether a different CAN port on at least one end of the connection with the 6XD makes a difference or not.
again starting with the initial arrangement, replace the short cable between 3HC3 and 3HC4 with the 15m one. The result might indicate a problem with either the overall length of the CAN bus, or the position of the long section within the chain of boards.

For both tests, would you mind to post the M122 reports (if available)?

developeralgo222

@infiniteloop said in Duet 3HC Expansion looses Connection:

@developeralgo222

Decided to retest the isolated boards (Test bench setup )

Thank you for the tests. Looks like some meters of CAN more or less make a difference - that’s not OK at all. Could you please run two quick tests on the bench, both starting with the setup as on the photo:

physically swap 3HC boards #1 and #2 in the CAN chain, take the 15m cable to link the 6XD with 3HC2 (chaining 6XD -> 3HC2 -> 3HC1 -> 3HC3 -> 3HC4). No change of IDs. This is to see whether a different CAN port on at least one end of the connection with the 6XD makes a difference or not.

TEST 1

Same Test with 0.5m and 4m Works OK, Syncs and No errors
Same Test with 15m Kenable ADSL 2+ cable Does Not Work . Just reconnected errors and CAN timeout errors

......
1/28/2024, 6:08:17 PM	Error: Expansion board 3 reconnected
1/28/2024, 6:08:15 PM	Error: Expansion board 4 reconnected
.....
...
1/28/2024, 6:11:03 PM	

M122 B0

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode)
Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 153284
Dynamic ram: 117412 of which 0 recycled
Never used RAM 72440, free system stack 204 words
Tasks: NETWORK(1,ready,29.1%,172) ETHERNET(5,nWait,0.1%,323) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,795) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,342) MAIN(1,running,50.4%,444) IDLE(0,ready,20.4%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:05:22 ago, cause: power up
Last software reset at 2024-01-19 15:33, reason: User, Gcodes spinning, available RAM 71300, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 30.5, current 42.7, max 42.9
Supply voltage: min 23.9, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 265 queued, 265 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 2024-01-28 18:11:00
Slowest loop: 1200.90ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.8ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 1972, received 923, lost 0, errs 12242198, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 49), ts 1614/0/0
Tx timeouts 0,0,1613,58,0,298 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 7.24ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 1 0 0
Socket states: 5 5 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0


1/28/2024, 6:11:03 PM	Error: Expansion board 4 reconnected
1/28/2024, 6:11:00 PM	CAN response timeout: board 4, req type 6024, RID 58
                        Error: Expansion board 4 reconnected
                        Error: Expansion board 4 reconnected
1/28/2024, 6:10:59 PM	M122 B4
.........
.........
                        Error: Expansion board 4 reconnected
1/28/2024, 6:10:58 PM	Error: Expansion board 4 reconnected
............
1/28/2024, 6:10:56 PM	CAN response timeout: board 3, req type 6024, RID 57
                        Error: Expansion board 4 reconnected
                        Error: Expansion board 2 stopped sending status
1/28/2024, 6:10:54 PM	M122 B3
............
............
Error: Expansion board 4 reconnected
1/28/2024, 6:10:53 PM	Error: Expansion board 4 reconnected
......
......
1/28/2024, 6:10:52 PM	CAN response timeout: board 2, req type 6024, RID 56
1/28/2024, 6:10:50 PM	M122 B2
.......
1/28/2024, 6:10:50 PM	Error: Expansion board 4 reconnected
                        Error: Expansion board 4 reconnected
.......
1/28/2024, 6:10:49 PM	Error: Expansion board 2 stopped sending status
......
1/28/2024, 6:10:46 PM	M122 B1
                        CAN response timeout: board 1, req type 6024, RID 55

TEST 2:

again starting with the initial arrangement, replace the short cable between 3HC3 and 3HC4 with the 15m one. The result might indicate a problem with either the overall length of the CAN bus, or the position of the long section within the chain of boards.

For both tests, would you mind to post the M122 reports (if available)?

NOTE: i can confirm that we might have an Issue with CAN Bus cable Length . If this can be reproduced by someone else then its bad news.

Same Test with 0.5m and 4m Works OK, Syncs and No errors
Same Test with 15m Kenable ADSL 2+ cable Does Not Work ,

Where 6XD-->0.5m-->3HC1 -->0.5m-->3HC2-->0.5m-->3HC3-->15m-->3HC4

6XD-->0.5m-->3HC1 -->0.5m-->3HC2-->0.5m-->3HC3 All Sync except 3HC4 Which has 15m kenable ADSL 2+ cable connected between 3HC3 and 3HC4

1/28/2024, 6:34:08 PM	

M122 B4
CAN response timeout: board 4, req type 6024, RID 82


1/28/2024, 6:34:04 PM	

M122 B3
Diagnostics for board 3:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156832, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,6.0%,103) MAIN(1,running,92.8%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.2%,263), total 100.0%
Last reset 00:06:57 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 48852, writes 17 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 48853, writes 17 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 48854, writes 17 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -4/13, peak Rx sync delay 182, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.3, max 24.3
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 22.6C, current 29.6C, max 29.6C
Last sensors broadcast 0x00000000 found 0 148 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 3382, send timeouts 0, received 2066, lost 0, errs 14241581, boc 0, free buffers 38, min 38, error reg ffff00
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 6:34:01 PM	

M122 B2
Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156832, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,6.0%,103) MAIN(1,running,92.8%,393) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.2%,263), total 100.0%
Last reset 00:06:54 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 38393, writes 17 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 38393, writes 17 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 38394, writes 17 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -11/10, peak Rx sync delay 183, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.3
V12 voltage: min 12.3, current 12.3, max 12.4
MCU temperature: min 23.2C, current 30.8C, max 31.0C
Last sensors broadcast 0x00000000 found 0 128 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 3360, send timeouts 0, received 2052, lost 0, errs 14147821, boc 0, free buffers 38, min 38, error reg ffff00
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 6:33:59 PM	

M122 B1
Diagnostics for board 1:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 156904, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,6.0%,103) MAIN(1,running,92.8%,413) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.2%,263), total 100.0%
Last reset 00:06:52 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 20.0 steps/mm, standstill, SG min 0, mspos 8, reads 28357, writes 17 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 20.0 steps/mm, standstill, SG min 0, mspos 8, reads 28357, writes 17 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 20.0 steps/mm, standstill, SG min 0, mspos 8, reads 28358, writes 17 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -5/15, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.1, current 24.1, max 24.2
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 20.5C, current 26.3C, max 26.3C
Last sensors broadcast 0x00000000 found 0 16 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 3343, send timeouts 0, received 2042, lost 0, errs 14059073, boc 0, free buffers 38, min 38, error reg ffff00
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0


1/28/2024, 6:33:56 PM	

M122 B0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode)
Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 153284
Dynamic ram: 117916 of which 0 recycled
Never used RAM 71936, free system stack 202 words
Tasks: NETWORK(1,ready,35.2%,172) ETHERNET(5,nWait,0.1%,317) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,350) MAIN(1,running,64.4%,444) IDLE(0,ready,0.3%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:06:49 ago, cause: power up
Last software reset at 2024-01-19 15:33, reason: User, Gcodes spinning, available RAM 71300, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 28.3, current 42.7, max 42.7
Supply voltage: min 24.0, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 2024-01-28 18:33:55
Slowest loop: 5.37ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 2108, received 9884, lost 0, errs 13969212, boc 96
Longest wait 30ms for reply type 6018, peak Tx sync delay 1586, free buffers 50 (min 49), ts 2047/1998/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 7.36ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

I think further Testing with different Length using TEST 1 and 2 is required to rule out this issue of cable length. I have tried several self-made CAN Bus cables with different lengths around 4m to 5m and some work but some don't , i am assuming that my Network Cable craftmanship might not be the top notch ( BTW, i have always terminated all my own Network cables for over 25 years as an engineer). I assumed that Kenable ADSL 2+ cables should work in this case to rule out if its my cable termination work.

gloomyandy

@developeralgo222 Have you tested the long cable to just a single 3HC board (which must be the one with the terminator enabled - I assume the others do not have it enabled)?

developeralgo222

@gloomyandy said in Duet 3HC Expansion looses Connection:

@developeralgo222 Have you tested the long cable to just a single 3HC board (which must be the one with the terminator enabled - I assume the others do not have it enabled)?

Yes the Terminator jumpers are only on 3HC4 board. Just tested the long 15m Cable from 6XD --> 3HC4 only ( 2 boards ) and does not sync

M122 B4
CAN response timeout: board 4, req type 6024, RID 84

infiniteloop

@developeralgo222 Thank you for running all these tests and collecting the data. I’m not qualified to interpret the results in detail, but as you’ve eliminated all other potential suspects, it is now crystal clear that CAN is the single possible culprit. In my eyes, a great achievement and a major step towards a solution. Let's see what idea @DC42 comes up with. Maybe he wants you to run some more tests to pin down the root cause, so please keep up your test environment for the time being.