Duet 3HC Expansion looses Connection
-
@infiniteloop said in Duet 3HC Expansion looses Connection:
i know its frustrating for both me and the forum support.
True, frustration is the dominant feeling when I’m hunting for bugs in my hard- or software. Things become more complicated if large parts of the system are a terra incognita, i.e. unknown to me. Without better knowledge of your wiring (how you integrate the Duets into the machine), I can assume your setup to be "right" (which, in itself, may be debatable), or make other assumptions, but I can’t do without these unless I know and comprehend all potentially relevant facts. Sorry for the inconvenience, but I know of no other way to deal with errors that simply shouldn’t be there.
RE 2 or 3 cases:
This and next below are same test
In the context of the isolated stack, you posted a photo (see below, Jan 26th) with several sockets populated. That’s different from ”just CAN and VIN”. Whether this is relevant or not… well, that depends on my assumptions about your electrical layout. You are the only one who knows the PnP in detail.
Sorry for the confusion on the photos. Yes the photo was to show the isolation of the stack. It's not Powered at all in that photo. When i am testing nothing is connected apart from the VIN & CAN Bus cables. Its frustrating and i wish i could find the problem causing CAN Bus sync issue and not have to ask for help from the designers and support. i have tried every suggested solution or test and CAN Bus Synchronization has not worked as expected
NOTE: Just received the first batch of Cables today and i replaced the 6XD --> 3HC1 cable with 15m Kenable ADSL 2+ and that did not make any difference . No sync at all . As i thought before i don't think it's the Cables themselves. i am also still waiting for 3 short Kenable Cables to replace the CAN Bus connections between the 3HC boards. But i don't think that will change anything at the moment.
Decided to retest the isolated boards (Test bench setup ) by removing them from the PnP machine using kenable ADSL 2+ cable from 6XD ---> 3HC1 . It does not work. Stacked or Not does not matter it does not work. Tested individual 3HC boards they also do not sync with 6XD using Kenable ADSL 2+ cable
-
Tested Long 15m Kenable ADSL 2+ between 6XD --> 3HC1 ---- The test bench setup does not work
-
Tested self-made short Cable 0.5m (50cm) between 6XD --> 3HC1 ---- The test bench setup works and syncs without any errors so far
-
Tested self-made short Cable 4m (50cm) between 6XD --> 3HC1 ---- The test bench setup works and syncs without any errors so far
Now trying to test different lengths between 6XD --> 3HC1 on the isolated test bench . Frankly i am not sure what to make of this results . CAN Bus says a length upto 40m @ 1 Mb/s rate is fine
1/28/2024, 2:25:16 PM M122 B0 === Diagnostics === RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode) Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2 Used output buffers: 1 of 40 (29 max) === RTOS === Static ram: 153284 Dynamic ram: 117992 of which 0 recycled Never used RAM 71860, free system stack 202 words Tasks: NETWORK(1,ready,31.1%,182) ETHERNET(5,nWait,0.0%,321) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,351) MAIN(1,running,68.8%,444) IDLE(0,ready,0.0%,30), total 100.0% Owned mutexes: HTTP(MAIN) === Platform === Last reset 00:11:10 ago, cause: power up Last software reset at 2024-01-19 15:33, reason: User, Gcodes spinning, available RAM 71300, slot 0 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a Error status: 0x00 MCU temperature: min 28.0, current 43.3, max 43.4 Supply voltage: min 24.0, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0 Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0 Events: 0 queued, 0 completed Driver 0: ok Driver 1: ok Driver 2: ok Driver 3: ok Driver 4: ok Driver 5: ok Date/time: 2024-01-28 14:25:15 Slowest loop: 24.13ms; fastest: 0.07ms === Storage === Free file entries: 20 SD card 0 detected, interface speed: 25.0MBytes/sec SD card longest read time 2.3ms, write time 0.0ms, max retries 0 === Move === DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0 === DDARing 0 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 === GCodes === Movement locks held by null, null HTTP is ready with "M122 B0" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000000 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === CAN === Messages queued 3455, received 21664, lost 0, errs 217, boc 0 Longest wait 2ms for reply type 6024, peak Tx sync delay 252, free buffers 50 (min 49), ts 3353/3352/0 Tx timeouts 0,0,0,0,0,0 === Network === Slowest loop: 7.14ms; fastest: 0.03ms Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0) HTTP sessions: 1 of 8 = Ethernet = Interface state: active Error counts: 0 0 0 0 0 0 Socket states: 5 2 2 2 2 0 0 0 === Multicast handler === Responder is inactive, messages received 0, responses 0 1/28/2024, 2:25:12 PM M122 B4 Diagnostics for board 4: Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47) Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10) All averaging filters OK Never used RAM 156952, free system stack 202 words Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,399) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0% Last reset 00:11:06 ago, cause: power up Last software reset data not available Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43296, writes 11 timeouts 0, steps req 0 done 0 Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43297, writes 11 timeouts 0, steps req 0 done 0 Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 43297, writes 11 timeouts 0, steps req 0 done 0 Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00 Peak sync jitter -6/7, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled VIN voltage: min 24.3, current 24.3, max 24.4 V12 voltage: min 12.4, current 12.4, max 12.4 MCU temperature: min 22.9C, current 30.4C, max 30.6C Last sensors broadcast 0x00000000 found 0 38 ticks ago, 0 ordering errs, loop time 0 CAN messages queued 5357, send timeouts 0, received 3342, lost 0, errs 223, boc 0, free buffers 38, min 38, error reg ff0000 dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0 1/28/2024, 2:25:09 PM M122 B3 Diagnostics for board 3: Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47) Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25) All averaging filters OK Never used RAM 156832, free system stack 202 words Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0% Last reset 00:11:03 ago, cause: power up Last software reset data not available Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32406, writes 17 timeouts 0, steps req 0 done 0 Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32407, writes 17 timeouts 0, steps req 0 done 0 Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 32408, writes 17 timeouts 0, steps req 0 done 0 Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00 Peak sync jitter -5/7, peak Rx sync delay 183, resyncs 0/0, no timer interrupt scheduled VIN voltage: min 21.3, current 24.2, max 24.3 V12 voltage: min 12.3, current 12.3, max 12.4 MCU temperature: min 22.8C, current 30.0C, max 30.0C Last sensors broadcast 0x00000000 found 0 242 ticks ago, 0 ordering errs, loop time 0 CAN messages queued 5350, send timeouts 0, received 3345, lost 0, errs 223, boc 0, free buffers 38, min 38, error reg ff0000 dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0 1/28/2024, 2:25:06 PM M122 B2 Diagnostics for board 2: Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47) Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25) All averaging filters OK Never used RAM 156832, free system stack 202 words Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,421) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0% Last reset 00:11:01 ago, cause: power up Last software reset data not available Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21924, writes 17 timeouts 0, steps req 0 done 0 Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21924, writes 17 timeouts 0, steps req 0 done 0 Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 21925, writes 17 timeouts 0, steps req 0 done 0 Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00 Peak sync jitter -6/6, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled VIN voltage: min 24.2, current 24.2, max 24.3 V12 voltage: min 12.3, current 12.3, max 12.4 MCU temperature: min 22.8C, current 31.6C, max 31.6C Last sensors broadcast 0x00000000 found 0 66 ticks ago, 0 ordering errs, loop time 0 CAN messages queued 5330, send timeouts 0, received 3331, lost 0, errs 217, boc 0, free buffers 38, min 38, error reg ff0000 dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0 1/28/2024, 2:25:03 PM M122 B1 Diagnostics for board 1: Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47) Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25) All averaging filters OK Never used RAM 156904, free system stack 202 words Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.7%,103) MAIN(1,running,93.1%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0% Last reset 00:10:57 ago, cause: power up Last software reset data not available Driver 0: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59337, writes 0 timeouts 0, steps req 0 done 0 Driver 1: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59336, writes 0 timeouts 0, steps req 0 done 0 Driver 2: pos 0, 20.0 steps/mm, standstill, SG min n/a, mspos 8, reads 59336, writes 0 timeouts 0, steps req 0 done 0 Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00 Peak sync jitter -3/5, peak Rx sync delay 177, resyncs 0/0, no timer interrupt scheduled VIN voltage: min 24.1, current 24.1, max 24.1 V12 voltage: min 12.4, current 12.4, max 12.4 MCU temperature: min 19.6C, current 26.8C, max 26.8C Last sensors broadcast 0x00000000 found 0 83 ticks ago, 0 ordering errs, loop time 0 CAN messages queued 152, send timeouts 0, received 84, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0 dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0 -
-
Decided to retest the isolated boards (Test bench setup )
Thank you for the tests. Looks like some meters of CAN more or less make a difference - that’s not OK at all. Could you please run two quick tests on the bench, both starting with the setup as on the photo:
-
physically swap 3HC boards #1 and #2 in the CAN chain, take the 15m cable to link the 6XD with 3HC2 (chaining 6XD -> 3HC2 -> 3HC1 -> 3HC3 -> 3HC4). No change of IDs. This is to see whether a different CAN port on at least one end of the connection with the 6XD makes a difference or not.
-
again starting with the initial arrangement, replace the short cable between 3HC3 and 3HC4 with the 15m one. The result might indicate a problem with either the overall length of the CAN bus, or the position of the long section within the chain of boards.
For both tests, would you mind to post the M122 reports (if available)?
-
-
@infiniteloop said in Duet 3HC Expansion looses Connection:
Decided to retest the isolated boards (Test bench setup )
Thank you for the tests. Looks like some meters of CAN more or less make a difference - that’s not OK at all. Could you please run two quick tests on the bench, both starting with the setup as on the photo:
- physically swap 3HC boards #1 and #2 in the CAN chain, take the 15m cable to link the 6XD with 3HC2 (chaining 6XD -> 3HC2 -> 3HC1 -> 3HC3 -> 3HC4). No change of IDs. This is to see whether a different CAN port on at least one end of the connection with the 6XD makes a difference or not.
TEST 1
- Same Test with 0.5m and 4m Works OK, Syncs and No errors
- Same Test with 15m Kenable ADSL 2+ cable Does Not Work . Just reconnected errors and CAN timeout errors
...... 1/28/2024, 6:08:17 PM Error: Expansion board 3 reconnected 1/28/2024, 6:08:15 PM Error: Expansion board 4 reconnected ..... ... 1/28/2024, 6:11:03 PM M122 B0 === Diagnostics === RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode) Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2 Used output buffers: 1 of 40 (29 max) === RTOS === Static ram: 153284 Dynamic ram: 117412 of which 0 recycled Never used RAM 72440, free system stack 204 words Tasks: NETWORK(1,ready,29.1%,172) ETHERNET(5,nWait,0.1%,323) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,795) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,342) MAIN(1,running,50.4%,444) IDLE(0,ready,20.4%,30), total 100.0% Owned mutexes: HTTP(MAIN) === Platform === Last reset 00:05:22 ago, cause: power up Last software reset at 2024-01-19 15:33, reason: User, Gcodes spinning, available RAM 71300, slot 0 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a Error status: 0x00 MCU temperature: min 30.5, current 42.7, max 42.9 Supply voltage: min 23.9, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0 Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0 Events: 265 queued, 265 completed Driver 0: ok Driver 1: ok Driver 2: ok Driver 3: ok Driver 4: ok Driver 5: ok Date/time: 2024-01-28 18:11:00 Slowest loop: 1200.90ms; fastest: 0.07ms === Storage === Free file entries: 20 SD card 0 detected, interface speed: 25.0MBytes/sec SD card longest read time 2.8ms, write time 0.0ms, max retries 0 === Move === DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0 === DDARing 0 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 === GCodes === Movement locks held by null, null HTTP is ready with "M122 B0" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000000 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === CAN === Messages queued 1972, received 923, lost 0, errs 12242198, boc 0 Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 49), ts 1614/0/0 Tx timeouts 0,0,1613,58,0,298 last cancelled message type 30 dest 127 === Network === Slowest loop: 7.24ms; fastest: 0.03ms Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0) HTTP sessions: 1 of 8 = Ethernet = Interface state: active Error counts: 0 0 0 1 0 0 Socket states: 5 5 2 2 2 0 0 0 === Multicast handler === Responder is inactive, messages received 0, responses 0 1/28/2024, 6:11:03 PM Error: Expansion board 4 reconnected 1/28/2024, 6:11:00 PM CAN response timeout: board 4, req type 6024, RID 58 Error: Expansion board 4 reconnected Error: Expansion board 4 reconnected 1/28/2024, 6:10:59 PM M122 B4 ......... ......... Error: Expansion board 4 reconnected 1/28/2024, 6:10:58 PM Error: Expansion board 4 reconnected ............ 1/28/2024, 6:10:56 PM CAN response timeout: board 3, req type 6024, RID 57 Error: Expansion board 4 reconnected Error: Expansion board 2 stopped sending status 1/28/2024, 6:10:54 PM M122 B3 ............ ............ Error: Expansion board 4 reconnected 1/28/2024, 6:10:53 PM Error: Expansion board 4 reconnected ...... ...... 1/28/2024, 6:10:52 PM CAN response timeout: board 2, req type 6024, RID 56 1/28/2024, 6:10:50 PM M122 B2 ....... 1/28/2024, 6:10:50 PM Error: Expansion board 4 reconnected Error: Expansion board 4 reconnected ....... 1/28/2024, 6:10:49 PM Error: Expansion board 2 stopped sending status ...... 1/28/2024, 6:10:46 PM M122 B1 CAN response timeout: board 1, req type 6024, RID 55TEST 2:
- again starting with the initial arrangement, replace the short cable between 3HC3 and 3HC4 with the 15m one. The result might indicate a problem with either the overall length of the CAN bus, or the position of the long section within the chain of boards.
For both tests, would you mind to post the M122 reports (if available)?
NOTE: i can confirm that we might have an Issue with CAN Bus cable Length . If this can be reproduced by someone else then its bad news.
- Same Test with 0.5m and 4m Works OK, Syncs and No errors
- Same Test with 15m Kenable ADSL 2+ cable Does Not Work ,
Where 6XD-->0.5m-->3HC1 -->0.5m-->3HC2-->0.5m-->3HC3-->15m-->3HC4
- 6XD-->0.5m-->3HC1 -->0.5m-->3HC2-->0.5m-->3HC3 All Sync except 3HC4 Which has 15m kenable ADSL 2+ cable connected between 3HC3 and 3HC4
1/28/2024, 6:34:08 PM M122 B4 CAN response timeout: board 4, req type 6024, RID 82 1/28/2024, 6:34:04 PM M122 B3 Diagnostics for board 3: Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47) Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25) All averaging filters OK Never used RAM 156832, free system stack 194 words Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,6.0%,103) MAIN(1,running,92.8%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.2%,263), total 100.0% Last reset 00:06:57 ago, cause: power up Last software reset data not available Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 48852, writes 17 timeouts 0, steps req 0 done 0 Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 48853, writes 17 timeouts 0, steps req 0 done 0 Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 48854, writes 17 timeouts 0, steps req 0 done 0 Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00 Peak sync jitter -4/13, peak Rx sync delay 182, resyncs 0/0, no timer interrupt scheduled VIN voltage: min 24.2, current 24.3, max 24.3 V12 voltage: min 12.3, current 12.3, max 12.4 MCU temperature: min 22.6C, current 29.6C, max 29.6C Last sensors broadcast 0x00000000 found 0 148 ticks ago, 0 ordering errs, loop time 0 CAN messages queued 3382, send timeouts 0, received 2066, lost 0, errs 14241581, boc 0, free buffers 38, min 38, error reg ffff00 dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0 1/28/2024, 6:34:01 PM M122 B2 Diagnostics for board 2: Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47) Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25) All averaging filters OK Never used RAM 156832, free system stack 194 words Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,6.0%,103) MAIN(1,running,92.8%,393) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.2%,263), total 100.0% Last reset 00:06:54 ago, cause: power up Last software reset data not available Driver 0: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 38393, writes 17 timeouts 0, steps req 0 done 0 Driver 1: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 38393, writes 17 timeouts 0, steps req 0 done 0 Driver 2: pos 0, 8.9 steps/mm, standstill, SG min 0, mspos 8, reads 38394, writes 17 timeouts 0, steps req 0 done 0 Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00 Peak sync jitter -11/10, peak Rx sync delay 183, resyncs 0/0, no timer interrupt scheduled VIN voltage: min 24.2, current 24.2, max 24.3 V12 voltage: min 12.3, current 12.3, max 12.4 MCU temperature: min 23.2C, current 30.8C, max 31.0C Last sensors broadcast 0x00000000 found 0 128 ticks ago, 0 ordering errs, loop time 0 CAN messages queued 3360, send timeouts 0, received 2052, lost 0, errs 14147821, boc 0, free buffers 38, min 38, error reg ffff00 dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0 1/28/2024, 6:33:59 PM M122 B1 Diagnostics for board 1: Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47) Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25) All averaging filters OK Never used RAM 156904, free system stack 194 words Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,6.0%,103) MAIN(1,running,92.8%,413) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.2%,263), total 100.0% Last reset 00:06:52 ago, cause: power up Last software reset data not available Driver 0: pos 0, 20.0 steps/mm, standstill, SG min 0, mspos 8, reads 28357, writes 17 timeouts 0, steps req 0 done 0 Driver 1: pos 0, 20.0 steps/mm, standstill, SG min 0, mspos 8, reads 28357, writes 17 timeouts 0, steps req 0 done 0 Driver 2: pos 0, 20.0 steps/mm, standstill, SG min 0, mspos 8, reads 28358, writes 17 timeouts 0, steps req 0 done 0 Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00 Peak sync jitter -5/15, peak Rx sync delay 180, resyncs 0/0, no timer interrupt scheduled VIN voltage: min 24.1, current 24.1, max 24.2 V12 voltage: min 12.4, current 12.4, max 12.4 MCU temperature: min 20.5C, current 26.3C, max 26.3C Last sensors broadcast 0x00000000 found 0 16 ticks ago, 0 ordering errs, loop time 0 CAN messages queued 3343, send timeouts 0, received 2042, lost 0, errs 14059073, boc 0, free buffers 38, min 38, error reg ffff00 dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0 1/28/2024, 6:33:56 PM M122 B0 === Diagnostics === RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.2 (2023-12-14 10:33:00) running on Duet 3 MB6XD v1.01 or later (standalone mode) Board ID: 0JD2M-999AL-D25S0-7J9DA-3SJ6S-T43V2 Used output buffers: 1 of 40 (29 max) === RTOS === Static ram: 153284 Dynamic ram: 117916 of which 0 recycled Never used RAM 71936, free system stack 202 words Tasks: NETWORK(1,ready,35.2%,172) ETHERNET(5,nWait,0.1%,317) HEAT(3,nWait,0.0%,369) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,334) CanClock(7,delaying,0.0%,350) MAIN(1,running,64.4%,444) IDLE(0,ready,0.3%,30), total 100.0% Owned mutexes: HTTP(MAIN) === Platform === Last reset 00:06:49 ago, cause: power up Last software reset at 2024-01-19 15:33, reason: User, Gcodes spinning, available RAM 71300, slot 0 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a Error status: 0x00 MCU temperature: min 28.3, current 42.7, max 42.7 Supply voltage: min 24.0, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0 Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0 Events: 0 queued, 0 completed Driver 0: ok Driver 1: ok Driver 2: ok Driver 3: ok Driver 4: ok Driver 5: ok Date/time: 2024-01-28 18:33:55 Slowest loop: 5.37ms; fastest: 0.07ms === Storage === Free file entries: 20 SD card 0 detected, interface speed: 25.0MBytes/sec SD card longest read time 2.3ms, write time 0.0ms, max retries 0 === Move === DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0 === DDARing 0 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 === GCodes === Movement locks held by null, null HTTP is ready with "M122 B0" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000000 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === CAN === Messages queued 2108, received 9884, lost 0, errs 13969212, boc 96 Longest wait 30ms for reply type 6018, peak Tx sync delay 1586, free buffers 50 (min 49), ts 2047/1998/0 Tx timeouts 0,0,0,0,0,0 === Network === Slowest loop: 7.36ms; fastest: 0.03ms Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0) HTTP sessions: 1 of 8 = Ethernet = Interface state: active Error counts: 0 0 0 0 0 0 Socket states: 5 2 2 2 2 0 0 0 === Multicast handler === Responder is inactive, messages received 0, responses 0I think further Testing with different Length using TEST 1 and 2 is required to rule out this issue of cable length. I have tried several self-made CAN Bus cables with different lengths around 4m to 5m and some work but some don't , i am assuming that my Network Cable craftmanship might not be the top notch ( BTW, i have always terminated all my own Network cables for over 25 years as an engineer). I assumed that Kenable ADSL 2+ cables should work in this case to rule out if its my cable termination work.
-
@developeralgo222 Have you tested the long cable to just a single 3HC board (which must be the one with the terminator enabled - I assume the others do not have it enabled)?
-
@gloomyandy said in Duet 3HC Expansion looses Connection:
@developeralgo222 Have you tested the long cable to just a single 3HC board (which must be the one with the terminator enabled - I assume the others do not have it enabled)?
Yes the Terminator jumpers are only on 3HC4 board. Just tested the long 15m Cable from 6XD --> 3HC4 only ( 2 boards ) and does not sync
M122 B4 CAN response timeout: board 4, req type 6024, RID 84 -
@developeralgo222 Thank you for running all these tests and collecting the data. I’m not qualified to interpret the results in detail, but as you’ve eliminated all other potential suspects, it is now crystal clear that CAN is the single possible culprit. In my eyes, a great achievement and a major step towards a solution. Let's see what idea @DC42 comes up with. Maybe he wants you to run some more tests to pin down the root cause, so please keep up your test environment for the time being.
-
@developeralgo222 I'm wondering if the CAN termination is damaged on the 6XD, as there appears to be little consistency between connections with the 3HC boards. I doubt you have four bad 3HCs!
The circuit is very similar there compared to the 3HC:
The CAN 1 termination resistors are R133 and R134, but aren't marked on the silkscreen I think. They are behind the CAN 1 OUT RJ11 connector. Also check that the drill-out termination vias (CAN1_TERM) on the back of the board are still intact:
Do you have another mainboard you can test? I think if you did you might have already tried that by now...
If that doesn't help, I think we're at the point where we should swap out your 6XD under warranty.
Ian
-
@droftarts said in Duet 3HC Expansion looses Connection:
@developeralgo222 I'm wondering if the CAN termination is damaged on the 6XD, as there appears to be little consistency between connections with the 3HC boards. I doubt you have four bad 3HCs!
The circuit is very similar there compared to the 3HC:
The CAN 1 termination resistors are R133 and R134, but aren't marked on the silkscreen I think. They are behind the CAN 1 OUT RJ11 connector. Also check that the drill-out termination vias (CAN1_TERM) on the back of the board are still intact:
Do you have another mainboard you can test? I think if you did you might have already tried that by now...
If that doesn't help, I think we're at the point where we should swap out your 6XD under warranty.
Ian
I don’t have another 6XD that I can use for test. What’s the process of swapping the 6XD , do I send to spool3D ( where I bought it) or to Duet in UK ? -
@developeralgo222 said in Duet 3HC Expansion looses Connection:
I don’t have another 6XD that I can use for test. What’s the process of swapping the 6XD , do I send to spool3D ( where I bought it) or to Duet in UK ?
We'll get Spool3D to swap it out for you. Can you confirm that you've checked the CAN resistors on the 6XD?
Ian
-
@droftarts said in Duet 3HC Expansion looses Connection:
@developeralgo222 said in Duet 3HC Expansion looses Connection:
I don’t have another 6XD that I can use for test. What’s the process of swapping the 6XD , do I send to spool3D ( where I bought it) or to Duet in UK ?
We'll get Spool3D to swap it out for you. Can you confirm that you've checked the CAN resistors on the 6XD?
Ian
Yes CAN termination resistors at the back looked ok when i tested the CAN_H and CAN_L pins (118 Ohms ) but by closely looking at them on the back i don't see any actual 2 x SMT Resistors (R133 = 59 Ohms, R134 = 59 Ohms ) on the solder pads other than the pads being connected as you can see on the photo.
-
i need to have this swapped by Spool3D as soon as we can, if we can have a new one shipped express by tomorrow and i will express this defective 6XD to them express too. We have been troubleshooting this for over 4 weeks now and i am extremely behind on the PnP project. Should i contact Spool3D directly myself or one of Duet guys is going to get in touch with Spool3D today
-
Please send an email to warranty@duet3d.com and CC your reseller. Include a link to this forum thread and the details of your original purchase. You'll receive a reply with a form to fill out.
-
@Phaedrux said in Duet 3HC Expansion looses Connection:
Please send an email to warranty@duet3d.com and CC your reseller. Include a link to this forum thread and the details of your original purchase. You'll receive a reply with a form to fill out.
done
-
@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:
On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.
So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.
Ian
-
@droftarts said in Duet 3HC Expansion looses Connection:
@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:
On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.
So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.
Ian
I am not sure whether its the 6XD board itself or CAN Resistor termination. Either way it doesn't reliably work on Test Bench or PnP Machine. If its reliable and working correctly the Cable length should not matter on the Test bench or PnP Machine. Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .
The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.
-
@developeralgo222 said in Duet 3HC Expansion looses Connection:
@droftarts said in Duet 3HC Expansion looses Connection:
@developeralgo222 The CAN1 terminating resistors appear to be there, either side of C197, just behind the RJ11 socket:
On the back of the board, the drill-to-disconnect pads there are to allow you to disable the termination. I see yours are soldered over, but that shouldn't matter, they are connected, so CAN termination should be enabled.
So I don't think it is a physical board fault around termination. It's very confusing, as this was all working at one point in a test bench setup. Let's see what happens with a new mainboard.
Ian
I am not sure whether its the 6XD board itself or CAN Resistor termination. Either way it doesn't reliably work on Test Bench or PnP Machine. If its reliable and working correctly the Cable length should not matter on the Test bench or PnP Machine. Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .
The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.
-
Please can someone from Duet Support Team that deals with Warranty (Warranty@duet3d.com) send me WAC code as required. Spool3D have got back to me but not Duet Team. i can't complete the required form without the required code from Duet. Time is of great importance to our project.
-
@developeralgo222 said in Duet 3HC Expansion looses Connection:
Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .
Wiggling the wires causes disconnections? Are you sure the pins in the RJ11 jack on the 6XD aren't bent or contaminated? It's not the soldering on the pins of the RJ11 on the back of the board, I can see that's fine. Though you could check for continuity between any of the pins on the RJ11 socket.
The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.
Most people won't ever need to touch these. If they do, they only need to drill out the via on the back of the board. If they decide they want to replace the termination, they just put a solder bridge back in. Far easier than trying to resolder two SMT resistors the size of a grain of sand!
Ian
-
@droftarts said in Duet 3HC Expansion looses Connection:
@developeralgo222 said in Duet 3HC Expansion looses Connection:
Even on Test Bench just changing the positions of same connecting cables causes it to stop working completely .
Wiggling the wires causes disconnections? Are you sure the pins in the RJ11 jack on the 6XD aren't bent or contaminated? It's not the soldering on the pins of the RJ11 on the back of the board, I can see that's fine. Though you could check for continuity between any of the pins on the RJ11 socket.
The Design of the CAN resistors on 6XD is very confusing. Why not just put actual SMT resistors and have users remove them if they don't need them, since we know that CAN Bus needs termination on Both ends if its a chain ? But i will leave that to the Designers.
Yes i am sure of that. i have terminated all my network cables for the past 25 Years. I even decided that may be something was wrong with all my network cable termination on all my cables and bought Kenable ADSL 2+ cables that were recommended ( even though very expensive here in North America). Tested all Kenable ADSL 2+ cables and my own Cables for Continuity, Shorts, Miswires, Reversals/Crossed, Split pairs and Shielding with both Professional Fluke and Sperry Crimp and Test tools.
Even with Kenable ADLS 2+ it still did not work. So i can guarantee that its not the Cables
-
@developeralgo222 I didn't say it was the cables. I said check the RJ11 socket on the 6XD.
Ian
