Prints Pausing: Connection Interrupted/Unknown msg type 6053

p8blr

@jay_s_uk Thanks I'll work on that!

dc42

@p8blr that's very odd, you should only get that message if the Tool1LC board is running older firmware than the main board. Are you certain that you have received the error message since the tool board was upgraded to 3.4.4 ?

p8blr

I made a new wire and the same thing happened again. I suppose this new wire could be bad as well, but I'm skeptical of that. No error messages this time. Here's some photos of the lcd and web console. It appears that the whole Duet has been reset.

@dc42 said in Prints Pausing: Connection Interrupted/Unknown msg type 6053:

@p8blr that's very odd, you should only get that message if the Tool1LC board is running older firmware than the main board. Are you certain that you have received the error message since the tool board was upgraded to 3.4.4 ?

I've restarted this print several times and changed so many things, so I suppose I can't be 100% sure they were running the same version. But I know that as of now they are both on 3.4.4 and I'm still getting the pausing issue.

jay_s_uk

@p8blr when you get the reset next time can you get an output of M122 and M122 B20 once reconnected?

p8blr

@jay_s_uk

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.4 (2022-10-20 16:19:01) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-9P63L-DJ3S0-7JKDJ-3SJ6T-T8M7B
Used output buffers: 3 of 40 (13 max)
=== RTOS ===
Static ram: 152824
Dynamic ram: 97228 of which 0 recycled
Never used RAM 100620, free system stack 200 words
Tasks: NETWORK(ready,29.4%,260) ETHERNET(notifyWait,0.0%,463) HEAT(notifyWait,0.0%,322) Move(notifyWait,0.0%,351) CanReceiv(notifyWait,0.0%,773) CanSender(notifyWait,0.0%,335) CanClock(delaying,0.0%,340) TMC(notifyWait,8.1%,91) MAIN(running,62.5%,1101) IDLE(ready,0.0%,30), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:44:38 ago, cause: software
Last software reset at 2022-11-08 10:40, reason: MemoryProtectionFault iaccViol, GCodes spinning, available RAM 97052, slot 1
Software reset code 0x4163 HFSR 0x00000000 CFSR 0x00000001 ICSR 0x00400804 BFAR 0x00000000 SP 0x2041b500 Task MAIN Freestk 1670 ok
Stack: 00000006 20419a8c 20418198 00000000 000001e8 00489457 20419a8c 00030200 00000000 ffffffff 2041b59c 2042debc 00000000 00000001 00000000 00488f37 2042debc 00000000 00000000 0048927f 00000000 00489457 00000000 ffffffff 31612900 2042debc 20419a8c
Error status: 0x00
Aux0 errors 0,0,0
Step timer max interval 130
MCU temperature: min 39.6, current 40.5, max 42.1
Supply voltage: min 23.6, current 23.7, max 23.8, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 156, reads 35581, writes 14 timeouts 0
Driver 1: standstill, SG min 0, mspos 612, reads 35581, writes 14 timeouts 0
Driver 2: standstill, SG min 0, mspos 484, reads 35581, writes 14 timeouts 0
Driver 3: standstill, SG min 0, mspos 8, reads 35585, writes 11 timeouts 0
Driver 4: standstill, SG min 0, mspos 8, reads 35585, writes 11 timeouts 0
Driver 5: standstill, SG min 0, mspos 8, reads 35585, writes 11 timeouts 0
Date/time: 2022-11-08 11:24:49
Slowest loop: 11.62ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 5.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 24104, received 53585, lost 0, boc 0
Longest wait 3ms for reply type 6026, peak Tx sync delay 297, free buffers 50 (min 49), ts 13392/13391/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 5.77ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

M122 B20
Diagnostics for board 20:
Duet TOOL1LC rev 1.0 or earlier firmware version 3.4.4 (2022-10-14 11:46:33)
Bootloader ID: not available
All averaging filters OK
Never used RAM 3608, free system stack 17 words
Tasks: Move(notifyWait,0.2%,99) HEAT(notifyWait,0.9%,77) CanAsync(notifyWait,0.0%,55) CanRecv(notifyWait,0.1%,74) CanClock(notifyWait,0.1%,65) TMC(delaying,10.4%,57) MAIN(running,71.2%,351) IDLE(ready,0.0%,40) AIN(delaying,17.1%,142), total 100.0%
Last reset 02:14:10 ago, cause: power up
Last software reset data not available
Driver 0: pos 2776783, 410.0 steps/mm,standstill, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 25464, writes 14, timeouts 0, DMA errors 0, CC errors 0, steps req 3933647 done 3933647
Moves scheduled 7354, completed 7354, in progress 0, hiccups 0, step errors 0, maxPrep 707, maxOverdue 1, maxInc 1, mcErrs 0, gcmErrs 0
Peak sync jitter -1/5, peak Rx sync delay 602, resyncs 0/1, no step interrupt scheduled
VIN voltage: min 23.4, current 23.8, max 23.9
MCU temperature: min 25.8C, current 48.8C, max 56.9C
Last sensors broadcast 0x00000002 found 1 4 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 173810, send timeouts 0, received 79843, lost 0, free buffers 37, min 36, error reg 110000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 548, adv 35624/74660
Accelerometer: none
I2C bus errors 4619, naks 4619, other errors 4619

jay_s_uk

@p8blr looks like the reset is on the mainboard so hopefully @dc42 can shed some light on it

dc42

@p8blr the diagnostic data you posted suggests that the reset may have been caused by an event such as a voltage transient or static discharge. Extruding plastic builds up static discharge; so please ensure that the hot end metalwork is connected to Duet ground, either directly or through a resistor. The latest versions of the tool board have all four mounting holes connected to ground through a resistor, so the mounting screws can be used to ground the hot end; but as your tool board is running an old bootloader version, I don't think that applies to your board.

I also suggest you update the bootloader on your tool board. See https://docs.duet3d.com/User_manual/RepRapFirmware/Updating_bootloader.

p8blr

@dc42 I connected a chassis ground wire directly to the heaterblock. It is a 1.0 toolboard so I used the ground of the power cable instead. Lastly, I went ahead and updated the bootloader on the toolboard and resumed my print. I will update you with the results. Thanks.

p8blr

@dc42

@p8blr said in Prints Pausing: Connection Interrupted/Unknown msg type 6053:

@dc42 I connected a chassis ground wire directly to the heaterblock. It is a 1.0 toolboard so I used the ground of the power cable instead. Lastly, I went ahead and updated the bootloader on the toolboard and resumed my print. I will update you with the results. Thanks.

After trying these things, this morning I found my printer stopped again.

I don't see anything obvious on the LCD or console, other than the web interface console is blank again.

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.4 (2022-10-20 16:19:01) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-9P63L-DJ3S0-7JKDJ-3SJ6T-T8M7B
Used output buffers: 1 of 40 (13 max)
=== RTOS ===
Static ram: 152824
Dynamic ram: 97228 of which 0 recycled
Never used RAM 100620, free system stack 192 words
Tasks: NETWORK(ready,1022.7%,232) ETHERNET(notifyWait,0.3%,455) HEAT(notifyWait,0.8%,322) Move(notifyWait,0.1%,351) CanReceiv(notifyWait,1.7%,773) CanSender(notifyWait,0.0%,335) CanClock(delaying,0.4%,340) TMC(notifyWait,638.0%,91) MAIN(running,1033.0%,1101) IDLE(ready,0.0%,30), total 2697.1%
Owned mutexes:
=== Platform ===
Last reset 09:40:00 ago, cause: software
Last software reset at 2022-11-09 22:11, reason: MemoryProtectionFault mmarValid daccViol, GCodes spinning, available RAM 97028, slot 2
Software reset code 0x4163 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x00427804 BFAR 0x0000008a SP 0x2041b530 Task MAIN Freestk 1682 ok
Stack: 00000042 2042e250 2041b5a4 00000101 0000023c 00460be7 0045fb98 01030000 00000000 00489457 00000000 ffffffff 0045c58d 2042debc 20419a8c 2042e254 ffffffff 00000000 f25a41f8 0048956f 2042e250 2042aaa0 00000000 f25a41f8 a5a5a5a5 00460bbf 00000000
Error status: 0x00
Aux0 errors 0,0,0
Step timer max interval 131
MCU temperature: min 37.3, current 37.8, max 40.6
Supply voltage: min 23.6, current 23.7, max 23.8, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 212, reads 58623, writes 14 timeouts 0
Driver 1: standstill, SG min 0, mspos 236, reads 58623, writes 14 timeouts 0
Driver 2: standstill, SG min 0, mspos 428, reads 58623, writes 14 timeouts 0
Driver 3: standstill, SG min 0, mspos 8, reads 58626, writes 11 timeouts 0
Driver 4: standstill, SG min 0, mspos 8, reads 58626, writes 11 timeouts 0
Driver 5: standstill, SG min 0, mspos 8, reads 58626, writes 11 timeouts 0
Date/time: 2022-11-10 07:51:52
Slowest loop: 8.07ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 5.5ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 313205, received 696040, lost 0, boc 0
Longest wait 2ms for reply type 6031, peak Tx sync delay 274, free buffers 50 (min 49), ts 174004/174003/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 4.39ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

M122 B20
Diagnostics for board 20:
Duet TOOL1LC rev 1.0 or earlier firmware version 3.4.4 (2022-10-14 11:46:33)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 3608, free system stack 17 words
Tasks: Move(notifyWait,0.0%,99) HEAT(notifyWait,0.2%,75) CanAsync(notifyWait,0.0%,51) CanRecv(notifyWait,0.0%,74) CanClock(notifyWait,0.0%,65) TMC(notifyWait,3.0%,57) MAIN(running,91.9%,351) IDLE(ready,0.0%,40) AIN(delaying,4.9%,142), total 100.0%
Last reset 17:32:47 ago, cause: software
Last software reset data not available
Driver 0: pos 40872078, 410.0 steps/mm,standstill, SG min 2, read errors 0, write errors 0, ifcnt 70, reads 5692, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 119476, completed 119476, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 0/3, peak Rx sync delay 186, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 23.8, current 23.8, max 23.8
MCU temperature: min 39.5C, current 44.7C, max 53.4C
Last sensors broadcast 0x00000002 found 1 188 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 254, send timeouts 0, received 110, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: none
I2C bus errors 4493, naks 4493, other errors 4493

dc42

@p8blr thanks for the additional report. It's very different from the first one and again looks like it was caused by a transient rather than a software bug.

Do you have the USB port of the MB6HC connected to anything? If so then you may have a USB ground loop, see https://docs.duet3d.com/en/User_manual/Overview/USB_ground_loops.

p8blr

@dc42 I do not have USB connected. This printer has been running great for months up until now, and nothing has been changed. I have an oscilloscope, is there anything you'd like me to measure?

p8blr

@dc42 It paused again. I've swapped out the 6HC with a spare I had to see if that resolves the issue.

p8blr

@p8blr The printer paused sometime last night with the new 6HC. Since I've already tried replacing the cable, I assume the problem has to be the 1LC toolboard now...

p8blr

Update. I've swapped out both the duet 3 and toolboard and the printer paused again... I'm going to try swapping out the power supply next. This is very odd because I have two identical printers and one doesn't have this problem, so the power supply is the last variable I can think of...

dc42

@p8blr are you absolutely certain that when the printer paused this time, it gave the same message "Error: Board 20 received unknown msg type 6053." ?

p8blr

@dc42 No it hasn't given that messages any of the last half dozen or so times it's paused. I have no information on the screen, and the console is basically empty, leading me to believe it's being reset.

dc42

@p8blr said in Prints Pausing: Connection Interrupted/Unknown msg type 6053:

@dc42 No it hasn't given that messages any of the last half dozen or so times it's paused. I have no information on the screen, and the console is basically empty, leading me to believe it's being reset.

In that case, please post a M122 report for each board in your system.

p8blr

@dc42 I will once the printer stops again. After replacing the power supply, I removed all the extra grounding wires and restored the printer back to the original boards. It's been running for roughly 24 hours now, so I'm beginning to suspect the power supply was the issue. I'm going to put the suspected bad power supply in a different printer to confirm.

p8blr

@dc42 Quick update, the printer is still running, I have an estimated 37 hours remaining on the print right now. Installing a new power supply seems to have fixed the problem. What's odd is that the old power supply is installed in an identical printer and hasn't caused that one to fail yet... I guess this issue is solved, but I don't feel satisfied with the answer lol. I can still send you a M122 on Monday when the printer is finished if you'd like.