My printer reboots alone…. 3.5.1

dc42

To those of you experiencing this issue, please send the following command after power up, before you start printing:

M122 P500 S0

This will reduce performance a little, however it should change the imprecise Hard Fault resets that I see in your M122 reports to precise ones, which will make this easier for me to diagnose. Then post M122 reports if the problem happens again.

Aurimas

@dc42

I have enabled M122 P500 S0 but can't really see any difference in the output```
M122

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.3 (2024-01-24 17:56:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: PA279-4N6KL-K65J0-409N0-4Z02Z-HWYBD
Used output buffers: 2 of 40 (37 max)
=== RTOS ===
Static ram: 103200
Dynamic ram: 121928 of which 12 recycled
Never used RAM 16276, free system stack 198 words
Tasks: NETWORK(2,nWait 7,11.4%,212) HEAT(3,nWait 6,0.0%,326) Move(4,nWait 6,0.0%,312) CanReceiv(6,nWait 1,0.1%,773) CanSender(5,nWait 7,0.0%,337) CanClock(7,delaying,0.0%,349) TMC(4,delaying,1.4%,102) MAIN(1,running,86.2%,667) IDLE(0,ready,0.0%,30) AIN(4,delaying,0.8%,260), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:15:06 ago, cause: software
Last software reset at 2024-05-09 09:18, reason: HardFault invState, Gcodes spinning, available RAM 12104, slot 0
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fd8 Task NETW Freestk 483 ok
Stack: 00000180 00000002 200014e8 00000000 ffffffff 0009eeaf 00000000 600f0000 00000000 00000000 00000000 00000000 200305dc 00000800 20035b18 2002c3b8 200187dc 2002c216 200187dc 2001e9d8 0002fcf7 00000000 00000000 00000000 20012088 00000014 ffffffff
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 906550, completed 906549, timed out 0, errs 0
MCU temperature: min 29.1, current 29.5, max 29.9
Supply voltage: min 23.9, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 40, reads 42095, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 206, read errors 0, write errors 0, ifcnt 40, reads 42095, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 21, reads 42095, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 0, ifcnt 40, reads 42096, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 20, read errors 0, write errors 0, ifcnt 41, reads 42095, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 30, read errors 0, write errors 0, ifcnt 41, reads 42095, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 18, read errors 0, write errors 0, ifcnt 40, reads 42096, writes 0, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-05-09 09:33:57
Cache data hit count 1609260691
Slowest loop: 9.87ms; fastest: 0.15ms
=== Storage ===
Free file entries: 19
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 1, completed 1, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters 4 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 4168, received 18525, lost 0, errs 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 269, free buffers 26 (min 26), ts 2316/2316/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 21.21ms; fastest: 0.07ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta6
MAC address e8:68:e7:e1:4e:a5
Module reset reason: Power up, Vcc 3.41, flash size 2097152, free heap 40932
WiFi IP address 192.168.1.110
Signal strength -59dBm, channel 2, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Aurimas

@Phaedrux not sure why you moved it to Beta section.
the issue is happening with the official release and not the Beta

Phaedrux

@Aurimas said in My printer reboots alone…. 3.5.1:

the issue is happening with the official release and not the Beta

Bugs in the current release will get addressed in the coming betas.

Aurimas

@dc42 look at the output I got. Printer didn't reset, but was super jerky and was pausing when moving diagonally.
it is CORE XY
after rebooting it went back to normal
look at the tasks TOTAL and also MOVES section

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.1 (2024-04-19 14:41:25) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 44BBP-1N6KL-K65J0-409N2-1BW1Z-RD9N1
Used output buffers: 3 of 40 (40 max)
=== RTOS ===
Static ram: 103232
Dynamic ram: 123388 of which 12 recycled
Never used RAM 10584, free system stack 112 words
Tasks: NETWORK(2,nWait 7,486.2%,174) HEAT(3,nWait 6,8.4%,326) Move(4,nWait 6,426.8%,235) CanReceiv(6,nWait 1,29.9%,564) CanSender(5,nWait 7,9.7%,327) CanClock(7,delaying,2.3%,348) TMC(4,nWait 6,242.4%,68) MAIN(1,running,275.2%,641) IDLE(0,ready,197.6%,30) AIN(4,delaying,251.9%,256), total 1930.5%
Owned mutexes:
=== Platform ===
Last reset 47:53:44 ago, cause: software
Last software reset at 2024-05-08 17:20, reason: HardFault invState, Gcodes spinning, available RAM 11680, slot 2
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fd8 Task NETW Freestk 483 ok
Stack: 0000017f 00000002 200014e8 00000000 2003312b 0009eeaf 00000000 600f0000 00000000 00000000 00000000 00000000 2003073c 00000800 20035c78 2002c3b8 200187dc 2002c216 200187dc 2001e9d8 0002fcf7 00000000 00000000 00000000 20012088 00000014 244a4226
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 172431441, completed 172431441, timed out 0, errs 0
MCU temperature: min 23.4, current 25.1, max 38.4
Supply voltage: min 24.2, current 24.4, max 24.5, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/1408/1408, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 232, reads 23715, writes 44, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 232, reads 23714, writes 44, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x72
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 4, reads 23711, writes 48, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 4, reads 23710, writes 48, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 107, reads 23745, writes 14, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2024-05-11 07:41:20
Cache data hit count 4294967295
Slowest loop: 314.69ms; fastest: 0.10ms
=== Storage ===
Free file entries: 19
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 16.6ms, write time 207.9ms, max retries 0
=== Move ===
DMs created 83, segments created 56, maxWait 44492174ms, bed compensation in use: mesh, height map offset 0.071, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 95159, on retry 26820, too short 101316, wrong shape 1560516, maybepossible 24107
=== DDARing 0 ===
Scheduled moves 128820, completed 128820, hiccups 0, stepErrors 0, LaErrors 0, Underruns [18, 0, 66], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000807
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
check 0 clear 0
Extruder 1: no data received, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
=== CAN ===
Messages queued 3506442, received 6984385, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 498, free buffers 26 (min 24), ts 862123/862122/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 310.39ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1.0
MAC address c4:5b:be:ce:9c:81
Module reset reason: Power up, Vcc 3.36, flash size 2097152, free heap 43040
WiFi IP address 192.168.1.99
Signal strength -52dBm, channel 2, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Aurimas

@dc42 @gloomyandy @Phaedrux

guys got a question - does rolling back firmware completelly reinstalls it? I am having issues with 3 machines that worked find on RC2-RC4, but since I have installed 3.5.1 all of them have issues - reboots, overloads and so on. none of the machines work reliablly anymore.

droftarts

@Aurimas said in My printer reboots alone…. 3.5.1:

guys got a question - does rolling back firmware completelly reinstalls it?

Yes, it does, so long as all CAN connected boards and/or SBC are downgraded to the same firmware version.

I am having issues with 3 machines that worked find on RC2-RC4, but since I have installed 3.5.1 all of them have issues - reboots, overloads and so on. none of the machines work reliablly anymore.

There’s very little difference between rc4 and 3.5.1. It’s mostly some small changes around multiple motion systems, but this can affect the functioning of macros. Without some user testing between versions, it’s impossible for us to account for all the wide and varied implementations of RRF. So please bear with us while we resolve issues. The alternative is that you stick with whatever version works, and don’t get the benefit of newer versions, and we never know that a change or bug causes an issue for a certain group of users.

Ian

Aurimas

@droftarts that is the biggest problem - all versions are now playing up. this all started after I went to 3.5.1 and rolling back to older versions make no difference.
Until then all machines worked non stop didn't have a single issue.
I ran Rc3 and RC4 before upgrade to 3.5.1
after upgrade to 3.5.1 started getting issues with constant restarts.
rolled back to RC4 - same issues
rolled back to RC3 - still having issues and even RC2 is not helping.
All RC versions prior to this worked just fine.
now it appears that one of the machines MCU is being overloaded - the load goes to 1930% as you can see per log.
this hapenned multiples times. in this case movement is only 150mm/s
dropping the speed to 60mm/s helps to reduce the load, but the movement is still jittery and not acceptble.
this never been an issue prior to the update.
this is why I am asking if the installation of the firmware does clean install and should I try to downgrade via USB cable.
I have 3 machines that are no behaving any more. it is really upsetting when the printer crashes on a print that was going on for nearly 80 hours and had only 6 hours to go. not to mention kilograms of plastic wasted

gloomyandy

@Aurimas On the printers that you have downgraded to an earlier version are you still running with:

M122 P500 S0

set?

Aurimas

@gloomyandy

yes - I believe it was on.
even though after all these restarts I might have lost the track
here is another crash report

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.1 (2024-04-19 14:41:25) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: PA279-4N6KL-K65J0-409N0-4Z02Z-HWYBD
Used output buffers: 20 of 40 (36 max)
=== RTOS ===
Static ram: 103232
Dynamic ram: 122532 of which 12 recycled
Never used RAM 15640, free system stack 182 words
Tasks: NETWORK(2,nWait 7,14.4%,208) HEAT(3,nWait 6,0.0%,326) Move(4,nWait 6,0.0%,310) CanReceiv(6,nWait 1,0.1%,772) CanSender(5,nWait 7,0.0%,336) CanClock(7,delaying,0.0%,348) TMC(4,nWait 6,1.4%,102) MAIN(1,running,83.1%,651) IDLE(0,ready,0.0%,30) AIN(4,delaying,0.9%,260), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 39:07:39 ago, cause: software
Last software reset at 2024-05-11 16:34, reason: HardFault bfarValid precise, Gcodes spinning, available RAM 11100, slot 1
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x0047d803 BFAR 0x0000001c SP 0x20011ff0 Task NETW Freestk 483 ok
Stack: 2002c610 20030848 200014e8 00000000 2003322f 0002ff99 0002feac 610f0000 0002ff85 00000000 00000000 00000000 20030854 00000800 2002c5f8 2002c5f8 00000002 2002c41e 200187fc 2001e9f8 0003011f 00000000 00000000 00000000 200120a0 00000014 46b8173e
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 140859060, completed 140859060, timed out 0, errs 0
MCU temperature: min 22.9, current 23.1, max 23.5
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 0, ifcnt 81, reads 62161, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 106, read errors 0, write errors 0, ifcnt 81, reads 62161, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 45, reads 62161, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 64, read errors 0, write errors 0, ifcnt 83, reads 62161, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 66, read errors 0, write errors 0, ifcnt 84, reads 62161, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 56, read errors 0, write errors 0, ifcnt 84, reads 62161, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 68, read errors 0, write errors 0, ifcnt 83, reads 62161, writes 0, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-05-13 07:41:49
Cache data hit count 4294967295
Slowest loop: 32.79ms; fastest: 0.16ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 3.7ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 1, completed 1, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters 4 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 6162, received 27379, lost 0, errs 0, boc 0
Longest wait 6ms for reply type 6024, peak Tx sync delay 261, free buffers 26 (min 25), ts 3419/3419/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 20.35ms; fastest: 0.06ms
Responder states: MQTT(0) HTTP(1) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta6
MAC address e8:68:e7:e1:4e:a5
Module reset reason: Power up, Vcc 3.41, flash size 2097152, free heap 33128
WiFi IP address 192.168.1.110
Signal strength -56dBm, channel 2, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 3 0 0 0 0 0

CorvoConV

@Aurimas I have just now the same reset reason just now....

gloomyandy

@CorvoConV Please see https://forum.duet3d.com/post/338744 if you were running with M122 P500 S0 set please post the output from M122.

dc42

@CorvoConV @Aurimas thanks for reporting this and for posting M122 reports.

The reports you have provided suggest that you are seeing the same issue as this one https://github.com/Duet3D/RepRapFirmware/issues/935. The crash location is slightly different but there are a number of common factors:

Duet 3 Mini WiFi board
Network task has crashed
A network transaction has probably taken place just before the crash
The crash appears to be have been caused by a zero word having been written to a value on the stack

To help me make further connections, please report the list of preferred languages you have selected in your browser. If you are running Chrome then you can get this information by going to Settings, then Languages, and you will see the list in the Preferred Languages box.

PS - the strange values in the task total field of the M122 report are not an issue. The value overflows if M122 hasn't been run for more than about 90 minutes since the machine was started or M122 was last run. I will fix this in the next release.

dc42 created this issue in Duet3D/RepRapFirmware

closed Regular crashes on Duet 3 Mini WiFi #935

DonStauffer

@dc42 FWIW I think this happens on my Duet 2 WiFi.

Exerqtor

@dc42 said in My printer reboots alone…. 3.5.1:

@CorvoConV @Aurimas thanks for reporting this and for posting M122 reports.

The reports you have provided suggest that you are seeing the same issue as this one https://github.com/Duet3D/RepRapFirmware/issues/935. The crash location is slightly different but there are a number of common factors:

Duet 3 Mini WiFi board

Network task has crashed

A network transaction has probably taken place just before the crash

The crash appears to be have been caused by a zero word having been written to a value on the stack

Somewhat relieving that this looks like it's manifesting on others as well. I haven't had any of these issues after i started to mainly use OrcaSlicer for DWC connections (no DWC tabs open in chrome on the desktop), with the occational Chrome(android) connection through my phone to check in on a ongoing print.

gloomyandy

@DonStauffer said in My printer reboots alone…. 3.5.1:

Last software reset at 2024-05-14 20:51, reason: AssertionFailed, Gcodes spinning, available RAM 920, slot 0

That looks like a different problem to me than the ones being reported here. What is happening when you have this problem? @dc42 may be able to confirm and may have a better idea what is causing the assert.

dc42

@DonStauffer I've moved your post to a new topic because it is not the same issue. See https://forum.duet3d.com/topic/35722/duet-2-assertion-failure-rrf-3-5-1.

DonStauffer

@gloomyandy The printer just reboots unexpectedly. It never used to happen, but now it happens a couple times per day, starting around the time I upgraded from 3.4.1 (I think; whatever the last full release was).

dc42

All, please see https://forum.duet3d.com/topic/36689/random-hard-fault-resets-on-duet-3-mini-wifi and provide any additional reports there. I am locking this thread.