Reboots/resets randomly - RRF 3.5.0-b4
-
@dc42
Still not a single crash with 3.5b2, so i'll go out on a limb and claim that it's a 3.5b3/b4 issue and not HW. -
@Exerqtor can you still get it to crash when you run 3.5.0-beta.4 ?
It could still be a hardware issue, because the memory layout changes between firmware versions. If it's a hardware issue then it might be benign in older firmware versions, e.g. if memory still gets corrupted but that memory is an unused area of the stack instead of a recently-used area.
If it was a a firmware issue, then I would expect to be able to reproduce the reset on a genuine Duet using your build configuration; but I ran it twice for 8 hours at a time and it didn't crash. I would also expect other users to report it.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42
Gotcha! I'll update to b4 again tomorrow and see how it plays out. If it's still an issue i simply have to run b2 until I get a genuine Duet 3 Mini. -
Updated to 3.5b4 yesterday evening(fresh binaries from github) , and the crashing started again.
M122 === Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode) Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL Used output buffers: 1 of 40 (38 max) === RTOS === Static ram: 102996 Dynamic ram: 124204 of which 12 recycled Never used RAM 11988, free system stack 182 words Tasks: NETWORK(1,ready,140.3%,219) HEAT(3,nWait,0.2%,327) Move(4,nWait,0.0%,348) CanReceiv(6,nWait,0.4%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.1%,351) TMC(4,nWait,8.9%,108) MAIN(1,running,260.0%,704) IDLE(0,ready,6.7%,29) AIN(4,delaying,6.1%,266), total 422.7% Owned mutexes: WiFi(NETWORK) === Platform === Last reset 03:40:27 ago, cause: software Last software reset at 2023-07-25 06:15, reason: HardFault invState, Platform spinning, available RAM 11988, slot 1 Software reset code 0x4060 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok Stack: 000001b0 00000002 200014ec 00000000 ffffffff 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 20032174 00000800 20035e68 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 00000000 Error status: 0x00 Aux0 errors 0,0,0 MCU revision 3, ADC conversions started 13227701, completed 13227701, timed out 0, errs 0 MCU temperature: min 35.5, current 36.1, max 39.4 Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 99/29, heap memory allocated/used/recyclable 2048/1884/1488, gc cycles 1629 Events: 0 queued, 0 completed Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 216, reads 22838, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 216, reads 22837, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 234, reads 22837, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 234, reads 22837, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 232, reads 22838, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 76, reads 22839, writes 10, timeouts 2, DMA errors 0, CC errors 0, failedOp 0x41 Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 76, reads 22840, writes 10, timeouts 0, DMA errors 0, CC errors 0 Date/time: 2023-07-25 09:55:34 Cache data hit count 4294967295 Slowest loop: 24.26ms; fastest: 0.14ms === Storage === Free file entries: 18 SD card 0 detected, interface speed: 22.5MBytes/sec SD card longest read time 4.5ms, write time 4.5ms, max retries 0 === Move === DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled === DDARing 0 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 Heater 1 is on, I-accum = 0.0 === GCodes === Movement locks held by null, null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is doing "G4 P100 " in state(s) 0 0, running macro Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000803 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === CAN === Messages queued 119057, received 264597, lost 0, boc 0 Longest wait 4ms for reply type 6053, peak Tx sync delay 274, free buffers 18 (min 17), ts 66139/66138/0 Tx timeouts 0,0,0,0,0,0 === Network === Slowest loop: 22.45ms; fastest: 0.00ms Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) HTTP sessions: 2 of 8 = WiFi = Interface state: active Module is connected to access point Failed messages: pending 0, notready 0, noresp 0 Firmware version 2.1beta4 MAC address bc:ff:4d:e6:b1:62 Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42968 WiFi IP address 192.168.10.50 Signal strength -54dBm, channel 1, mode 802.11n, reconnections 0 Clock register 00002001 Socket states: 0 0 0 0 0 0 0 0 -
@Exerqtor was the machine idle or printing?
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42
Idle. It's been a couple more crashes since last post to, also while idle.M122 === Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode) Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL Used output buffers: 19 of 40 (28 max) === RTOS === Static ram: 102996 Dynamic ram: 124180 of which 12 recycled Never used RAM 12012, free system stack 186 words Tasks: NETWORK(2,nWait,18.4%,219) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,662) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,delaying,1.2%,108) MAIN(1,running,78.5%,704) IDLE(0,ready,0.9%,29) AIN(4,delaying,0.8%,266), total 100.0% Owned mutexes: WiFi(NETWORK) === Platform === Last reset 00:18:41 ago, cause: software Last software reset at 2023-07-25 20:51, reason: HardFault invState, Gcodes spinning, available RAM 11236, slot 1 Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok Stack: 000001b0 00000002 200014ec 00000000 ffffffff 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 20032174 00000800 20035e68 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 b5ddc1d1 Error status: 0x00 Aux0 errors 0,0,0 MCU revision 3, ADC conversions started 1121100, completed 1121100, timed out 0, errs 0 MCU temperature: min 36.3, current 36.9, max 41.8 Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 99/29, heap memory allocated/used/recyclable 2048/1692/1296, gc cycles 137 Events: 0 queued, 0 completed Driver 0: standstill, SG min 174, read errors 0, write errors 1, ifcnt 103, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 1: standstill, SG min 16, read errors 0, write errors 1, ifcnt 103, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 111, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 110, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 108, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 175, reads 36350, writes 10, timeouts 0, DMA errors 0, CC errors 0 Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 175, reads 36350, writes 10, timeouts 0, DMA errors 0, CC errors 0 Date/time: 2023-07-25 21:10:08 Cache data hit count 1876944189 Slowest loop: 8.00ms; fastest: 0.14ms === Storage === Free file entries: 18 SD card 0 detected, interface speed: 22.5MBytes/sec SD card longest read time 4.3ms, write time 2.2ms, max retries 0 === Move === DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled === DDARing 0 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 Heater 1 is on, I-accum = 0.0 === GCodes === Movement locks held by null, null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is doing "G4 P100 " in state(s) 0 0, running macro Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000803 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === CAN === Messages queued 10098, received 22458, lost 0, boc 0 Longest wait 3ms for reply type 6053, peak Tx sync delay 261, free buffers 18 (min 17), ts 5606/5605/0 Tx timeouts 0,0,0,0,0,0 === Network === Slowest loop: 10.73ms; fastest: 0.00ms Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) HTTP sessions: 2 of 8 = WiFi = Interface state: active Module is connected to access point Failed messages: pending 0, notready 0, noresp 0 Firmware version 2.1beta4 MAC address bc:ff:4d:e6:b1:62 Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 43124 WiFi IP address 192.168.10.50 Signal strength -52dBm, channel 1, mode 802.11n, reconnections 0 Clock register 00002001 Socket states: 0 0 0 0 0 0 0 0 -
@Exerqtor can you try un-configuring some devices to see whether that stops the board from crashing when it is sitting idle. For example, you could un-configure everything on the expansion boards; or un-configure all temperature sensors.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42
Sure, i'm up for trying. What would be the best approach to do this in your opinion (i'm not sure how to "un-configure" it once it's been configured
And would you suggest putting it in daemon or just make it a custom config.g to be ran as a test with mentioned stuff not configured and let the machine stay idle to see if it mitigates the crashing?
-
@Exerqtor I suggest temporarily commenting out the corresponding lines in config.g and rebooting.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
TLDR: Can a bad / "EOL" SD-card have been at fault for the issues I've been having?
@dc42
Never got around to change the lines since the printer was running stable on 3.5b2 and i had some prints going.I went through with buying a genuine Mini 5+ instead since i was/am in the impression that issue was most likely HW related. It still haven't arrived so I cant confirm that or not.
BUT on sunday evening while doing some routine maintenance (printer powered down, and power cord pulled) something weird happened when I finished up and booted the printer. It started all normally and i went through to home it but then PD promted me that i was missing all the homing files (one after one once i tried axis by axis), i went forrward to try turning on the chamber lights with it's macro: same issue, it's reported missing as well.
Baffeled i tried heating up the hotend to 75c just to see if the hotend fan turned on or not, it did. Turned off the heater and let it settle to where the fan turned off again. Started to think something must have gone wrong in the bootup (or something), so i powet toggeled the printer, this time PD warned about some JSON error while booting then just said "connecting" in top right corner and everything was unresponsive, couldn't connect to DWC either.
That's when i started to suspect something was going on with the SD card, powered down again and pulled the card out and put it in my computer, and sure AF it was corrupted with only a handfull of files readable, and ALL the files in
sys/and subfolders had changed names and exentions to something totally nonsense and went corrupted despite changing them to the right file extension.Tried running some recovery software on it, and managed to pull everything from
filaments/,firmware/& most of the job files ingcode/except one file. So yesterday i bought a new namebrand class 10 sdcard and started rebuilding the printer setup on the new card. And i just managed to boot and run the checks before going to bed.With everything confirmed working it started to dawn on me that maybe a shoddy SD card might have been the culprit for this reboot issue to, so I got back up from bed and flashed 3.5b4 to both the MB and Toolboard and ran a new set of initial checks to see that everything was fine (which it was) and went back to bed.
Checked the printer now after the morning walk with the dog, and still bo reboots/crashes/resets.
SOOOO can a shoddy SD card have caused these issues all along? Or is the corrupted card a totally standalone problem? -
@Exerqtor the only way I can think of that a corrupt SD card could cause the problem would be if RRF read a corrupt command from it that caused RRF to overflow an array bound, or something like that.
All the code I added to RRF does check that commands don't cause array bounds to be exceeded, but there might be some older code that doesn't.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 Ok, i to doubt the SD card was the culprit but i thought i should air the theory since it happened.
Still running 3.5b4 in idle at home, so i'm curious if it's gotten any new resets by the time i finish work
🤷♂️Curious to see if the behaviour changes with the new board to ofc.
-
Just had a reset now, so that comfirms it not being the SD-card lol.
M122 === Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode) Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL Used output buffers: 1 of 40 (28 max) === RTOS === Static ram: 102996 Dynamic ram: 125644 of which 12 recycled Never used RAM 10548, free system stack 182 words Tasks: NETWORK(1,ready,17.1%,203) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,1.2%,108) MAIN(1,running,80.6%,704) IDLE(0,ready,0.2%,29) AIN(4,delaying,0.8%,266), total 100.0% Owned mutexes: WiFi(NETWORK) === Platform === Last reset 00:44:41 ago, cause: software Last software reset at 2023-08-15 18:10, reason: HardFault invState, Gcodes spinning, available RAM 9052, slot 0 Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok Stack: 000001af 00000002 200014ec 00000000 2003338b 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 2003095c 00000800 20035e98 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 b5dd60fa Error status: 0x00 Aux0 errors 0,0,0 MCU revision 3, ADC conversions started 2681568, completed 2681568, timed out 0, errs 0 MCU temperature: min 36.1, current 36.9, max 42.2 Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/564/128, gc cycles 125 Events: 0 queued, 0 completed Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 126, reads 47121, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 1: standstill, SG min 14, read errors 0, write errors 1, ifcnt 126, reads 47121, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 98, reads 47121, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 99, reads 47120, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 99, reads 47121, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 47124, writes 10, timeouts 0, DMA errors 0, CC errors 0 Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 47124, writes 10, timeouts 0, DMA errors 0, CC errors 0 Date/time: 2023-08-15 18:55:09 Cache data hit count 4294967295 Slowest loop: 13.48ms; fastest: 0.13ms === Storage === Free file entries: 18 SD card 0 detected, interface speed: 22.5MBytes/sec SD card longest read time 5.3ms, write time 4.5ms, max retries 0 === Move === DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled === DDARing 0 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 Heater 1 is on, I-accum = 0.0 === GCodes === Movement locks held by null, null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is doing "G4 S1 " in state(s) 0 0, running macro Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000803 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === CAN === Messages queued 24140, received 53662, lost 0, boc 0 Longest wait 3ms for reply type 6053, peak Tx sync delay 316, free buffers 18 (min 17), ts 13407/13406/0 Tx timeouts 0,0,0,0,0,0 === Network === Slowest loop: 6.07ms; fastest: 0.00ms Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) HTTP sessions: 1 of 8 = WiFi = Interface state: active Module is connected to access point Failed messages: pending 0, notready 0, noresp 0 Firmware version 2.1beta4 MAC address bc:ff:4d:e6:b1:62 Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42972 WiFi IP address 192.168.10.50 Signal strength -57dBm, channel 1, mode 802.11n, reconnections 0 Clock register 00002001 Socket states: 0 0 0 0 0 0 0 0Hopefully the new board arrives tomorrow or tuesday, let's hope that solves it.
-
I had a reset as well, this one ANOTHER out of memory fault tied to the tool-board/CAN bus.
It's at the point I can't do any large prints without the machine resetting half way through and wasting filament. Incredibly frustrating when you have it running 1-2 days only to quit on you before it finishes. -
@Nuramori please try 3.5.0-rc.1.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 said in Reboots/resets randomly - RRF 3.5.0-beta4:
@Nuramori please try 3.5.0-rc.1.
Sorry for the lack of engagement but I haven't had time to fiddle with the printer in a while. I've been running 3.5.0-rc.1 since i got the new board. And it's still rebooting/crashing from time to time, but it's not been as often.
Here is the last few occations from the debug logg and a fresh M122 i just ran.
power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [info] Event logging stopped power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [debug] Done! power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running. power up + 00:00:04 [warn] WiFi module started power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50 2023-10-09 01:20:53 [warn] Date and time set at power up + 00:00:10 power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [info] Event logging stopped power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [debug] Done! power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running. power up + 00:00:04 [warn] WiFi module started power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50 2023-10-14 16:46:21 [warn] Date and time set at power up + 00:24:02 power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [info] Event logging stopped power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [debug] Done! power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running. power up + 00:00:04 [warn] WiFi module started power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50 2023-10-15 04:22:33 [warn] Date and time set at power up + 00:00:08 power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [info] Event logging stopped power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [debug] Done! power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running. power up + 00:00:04 [warn] WiFi module started power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50 2023-10-15 07:16:09 [warn] Date and time set at power up + 00:00:08 power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [info] Event logging stopped power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [debug] Done! power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running. power up + 00:00:04 [warn] WiFi module started power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50 2023-10-21 06:57:47 [warn] Date and time set at power up + 00:00:08 power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [info] Event logging stopped power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [debug] Done! power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running. power up + 00:00:04 [warn] WiFi module started power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50 2023-10-21 11:21:14 [warn] Date and time set at power up + 00:00:13M122:
M122 === Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1 (2023-08-31 16:16:56) running on Duet 3 Mini5plus WiFi (standalone mode) Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ Used output buffers: 1 of 40 (40 max) === RTOS === Static ram: 102836 Dynamic ram: 125300 of which 0 recycled Never used RAM 9800, free system stack 135 words Tasks: NETWORK(2,nWait,17.6%,233) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,243) CanReceiv(6,nWait,0.0%,773) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,0.7%,74) MAIN(1,running,80.5%,704) IDLE(0,ready,0.3%,29) AIN(4,delaying,0.8%,264), total 100.0% Owned mutexes: === Platform === Last reset 00:47:14 ago, cause: software Last software reset at 2023-10-21 11:20, reason: HardFault invState, Gcodes spinning, available RAM 13184, slot 2 Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok Stack: 000001af 00000002 200014ec 00000000 20032bfb 0009df2d 00000000 600f0000 00000000 00000000 00000000 00000000 20031a1c 00000800 20035710 2002bf30 20018668 2002bd9d 20018668 2001e880 0002fedf 00000000 00000000 00000000 20012058 00000014 0893d091 Error status: 0x04 Aux0 errors 0,1,0 MCU revision 3, ADC conversions started 2835341, completed 2835340, timed out 0, errs 0 MCU temperature: min 34.4, current 45.3, max 45.8 Supply voltage: min 22.4, current 24.0, max 26.6, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 99/34, heap memory allocated/used/recyclable 2048/964/508, gc cycles 134 Events: 0 queued, 0 completed Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 109, reads 18073, writes 43, timeouts 0, DMA errors 0, CC errors 0 Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 109, reads 18073, writes 43, timeouts 0, DMA errors 0, CC errors 0 Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 86, reads 18091, writes 25, timeouts 0, DMA errors 0, CC errors 0 Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 85, reads 18091, writes 24, timeouts 0, DMA errors 0, CC errors 0 Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 86, reads 18091, writes 25, timeouts 0, DMA errors 0, CC errors 0 Driver 5: not present Driver 6: not present Date/time: 2023-10-21 12:08:16 Cache data hit count 4294967295 Slowest loop: 29.65ms; fastest: 0.12ms === Storage === Free file entries: 18 SD card 0 detected, interface speed: 22.5MBytes/sec SD card longest read time 7.3ms, write time 28.1ms, max retries 0 === Move === DMs created 83, segments created 26, maxWait 1519826ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled Moves shaped first try 20, on retry 8, too short 0, wrong shape 0, maybepossible 0 === DDARing 0 === Scheduled moves 47, completed 47, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 11], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 Heater 0 is on, I-accum = 0.4 Heater 1 is on, I-accum = 0.0 === GCodes === Movement locks held by null, null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is doing "G4 S1" in state(s) 0 0, running macro Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000807 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === Filament sensors === Extruder 0 sensor: no filament === CAN === Messages queued 25544, received 58175, lost 0, boc 0 Longest wait 3ms for reply type 6013, peak Tx sync delay 258, free buffers 26 (min 25), ts 14175/14174/0 Tx timeouts 0,0,0,0,0,0 === Network === Slowest loop: 46.60ms; fastest: 0.00ms Responder states: MQTT(0) HTTP(2) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) HTTP sessions: 3 of 8 === WiFi === Interface state: active Module is connected to access point Failed messages: pending 0, notrdy 0, noresp 0 Firmware version 2.1beta4 MAC address c4:5b:be:ce:91:93 Module reset reason: Power up, Vcc 3.37, flash size 2097152, free heap 37884 WiFi IP address 192.168.10.50 Signal strength -50dBm, channel 6, mode 802.11n, reconnections 0 Clock register 00002001 Socket states: 0 0 0 0 0 0 0 0
Changing the thread title to reflect on the issue persisting to rc1 -
Hello! I started to have the same symptoms with my board (https://forum.duet3d.com/topic/33857/duet-2-wifi-hardfault-invstate) and I wonder if this tyoe of reset has something to do with the powersupply itself. Can a quick drop in the powersupply output result in this type of crash? A drop that is long enogh to cause a drop in the 3v3 circuit of the processor?
Just an idea.. -
Just for the sake of completeness (the other two issues I posted here are more pressing to me) - I have those reboots as well, running 3.5.0rc1 on a Duet 3 6HC with a Duet 3 wifi module on the latest firmware. In addition to that, I sometimes see connection losses of both the webinterface and the PanelDue that can only be cured by a reset.
Thankfully they did not happen during a print as of now, but rather when the printer has been sitting idle for an extended period of time. That might be caused by the fact that it usually takes the printer about 10+ hours to reset as far as I have noticed by now - all prints I did so far were well finished by then.
When this happens again, I will add an M122 printout here. So far,, the only thing I noticed on previous cases was the recurring note "gcodes spinning" as reset reason in all those cases.
-
"Glad" to see i'm not the only one having these crashes, must admit that.
I had another crash mid print roughly an hour ago
debug log:
power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [info] Event logging stopped power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [debug] Done! power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running. power up + 00:00:04 [warn] WiFi module started power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50 2023-10-21 22:24:09 [warn] Date and time set at power up + 00:00:09M122:
=== Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1 (2023-08-31 16:16:56) running on Duet 3 Mini5plus WiFi (standalone mode) Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ Used output buffers: 1 of 40 (40 max) === RTOS === Static ram: 102836 Dynamic ram: 124928 of which 0 recycled Never used RAM 10436, free system stack 136 words Tasks: NETWORK(1,ready,108.3%,228) HEAT(3,nWait,0.2%,335) Move(4,nWait,0.0%,261) CanReceiv(6,nWait,0.3%,773) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,342) TMC(4,nWait,4.3%,74) MAIN(1,running,529.8%,704) IDLE(0,ready,1.7%,29) AIN(4,delaying,5.3%,264), total 649.8% Owned mutexes: WiFi(NETWORK) === Platform === Last reset 01:52:48 ago, cause: software Last software reset at 2023-10-21 22:24, reason: HardFault invState, Gcodes spinning, available RAM 4720, slot 0 Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok Stack: 000001ae 00000002 200014ec 00000000 20032bfa 0009df2d 00000000 600f0000 00000000 00000000 00000000 00000000 200301d4 00000800 20035710 2002bf00 20018668 2002bd9d 20018668 2001e880 0002fedf 00000000 00000000 00000000 20012058 00000014 b5dd8a35 Error status: 0x04 Aux0 errors 0,0,0 MCU revision 3, ADC conversions started 6768325, completed 6768325, timed out 0, errs 0 MCU temperature: min 37.4, current 41.5, max 48.6 Supply voltage: min 2.2, current 24.1, max 25.2, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 99/34, heap memory allocated/used/recyclable 2048/1572/1116, gc cycles 318 Events: 0 queued, 0 completed Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 35, reads 28491, writes 35, timeouts 0, DMA errors 0, CC errors 0 Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 35, reads 28491, writes 35, timeouts 0, DMA errors 0, CC errors 0 Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 20, reads 28506, writes 20, timeouts 0, DMA errors 0, CC errors 0 Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 19, reads 28506, writes 19, timeouts 0, DMA errors 0, CC errors 0 Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 19, reads 28507, writes 19, timeouts 0, DMA errors 0, CC errors 0 Driver 5: not present Driver 6: not present Date/time: 2023-10-22 00:16:47 Cache data hit count 4294967295 Slowest loop: 13.37ms; fastest: 0.12ms === Storage === Free file entries: 18 SD card 0 detected, interface speed: 22.5MBytes/sec SD card longest read time 7.0ms, write time 4.6ms, max retries 0 === Move === DMs created 83, segments created 15, maxWait 5875833ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled Moves shaped first try 7, on retry 6, too short 0, wrong shape 0, maybepossible 0 === DDARing 0 === Scheduled moves 25, completed 25, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 Heater 1 is on, I-accum = 0.0 === GCodes === Movement locks held by null, null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is doing "G4 S1" in state(s) 0 0, running macro Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000807 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === Filament sensors === Extruder 0 sensor: no filament === CAN === Messages queued 60937, received 138751, lost 0, boc 0 Longest wait 2ms for reply type 6034, peak Tx sync delay 1839, free buffers 26 (min 25), ts 33842/33840/0 Tx timeouts 0,0,1,0,0,0 last cancelled message type 30 dest 127 === Network === Slowest loop: 8.41ms; fastest: 0.00ms Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) HTTP sessions: 1 of 8 === WiFi === Interface state: active Module is connected to access point Failed messages: pending 0, notrdy 0, noresp 0 Firmware version 2.1beta4 MAC address c4:5b:be:ce:91:93 Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42952 WiFi IP address 192.168.10.50 Signal strength -51dBm, channel 6, mode 802.11n, reconnections 0 Clock register 00002001 Socket states: 0 0 0 0 0 0 0 0 -
And another crash mid print, this is starting to become a REALLY annoying problem!
power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [info] Event logging stopped power up + 00:00:03 [info] Event logging started at level debug power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56) power up + 00:00:03 [debug] Done! power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running. power up + 00:00:04 [warn] WiFi module started power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50 2023-10-22 10:21:54 [warn] Date and time set at power up + 00:00:08=== Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1 (2023-08-31 16:16:56) running on Duet 3 Mini5plus WiFi (standalone mode) Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ Used output buffers: 3 of 40 (32 max) === RTOS === Static ram: 102836 Dynamic ram: 124104 of which 0 recycled Never used RAM 11620, free system stack 186 words Tasks: NETWORK(1,ready,17.3%,213) HEAT(3,nWait,0.0%,335) Move(4,nWait,0.0%,352) CanReceiv(6,nWait,0.0%,796) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,342) TMC(4,nWait,0.7%,108) MAIN(1,running,80.9%,704) IDLE(0,ready,0.3%,29) AIN(4,delaying,0.8%,264), total 100.0% Owned mutexes: WiFi(NETWORK) === Platform === Last reset 00:20:17 ago, cause: software Last software reset at 2023-10-22 10:21, reason: HardFault invState, Expansion spinning, available RAM 7500, slot 1 Software reset code 0x4072 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok Stack: 000001ae 00000002 200014ec 00000000 20032c0a 0009df2d 00000000 600f0000 00000000 00000000 00000000 00000000 20031a2c 00000800 2002bf00 2002bf00 00000001 2002bd9d 20018668 2001e880 0002fedf 00000000 00000000 00000000 20012058 00000014 b5dd8a35 Error status: 0x00 Aux0 errors 0,0,0 MCU revision 3, ADC conversions started 1217157, completed 1217157, timed out 0, errs 0 MCU temperature: min 41.0, current 41.7, max 48.4 Supply voltage: min 2.2, current 24.1, max 25.2, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/2008/1572, gc cycles 56 Events: 0 queued, 0 completed Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64030, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64030, writes 13, timeouts 0, DMA errors 0, CC errors 0 Driver 5: not present Driver 6: not present Date/time: 2023-10-22 10:42:02 Cache data hit count 2002338155 Slowest loop: 16.49ms; fastest: 0.13ms === Storage === Free file entries: 18 SD card 0 detected, interface speed: 22.5MBytes/sec SD card longest read time 7.1ms, write time 4.4ms, max retries 0 === Move === DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00 no step interrupt scheduled Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0 === DDARing 0 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === DDARing 1 === Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0 Heater 1 is on, I-accum = 0.0 === GCodes === Movement locks held by null, null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is doing "G4 S1" in state(s) 0 0, running macro Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 File2 is idle in state(s) 0 Queue2 is idle in state(s) 0 Q0 segments left 0, axes/extruders owned 0x0000803 Code queue 0 is empty Q1 segments left 0, axes/extruders owned 0x0000000 Code queue 1 is empty === Filament sensors === Extruder 0 sensor: no filament === CAN === Messages queued 10966, received 24933, lost 0, boc 0 Longest wait 2ms for reply type 6053, peak Tx sync delay 736, free buffers 26 (min 25), ts 6086/6084/0 Tx timeouts 0,0,1,0,0,0 last cancelled message type 30 dest 127 === Network === Slowest loop: 24.59ms; fastest: 0.00ms Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) HTTP sessions: 2 of 8 === WiFi === Interface state: active Module is connected to access point Failed messages: pending 0, notrdy 0, noresp 0 Firmware version 2.1beta4 MAC address c4:5b:be:ce:91:93 Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 43128 WiFi IP address 192.168.10.50 Signal strength -46dBm, channel 6, mode 802.11n, reconnections 0 Clock register 00002001 Socket states: 0 0 0 0 0 0 0 0Needless to say that the hypothesis of the core issue being a non Duet board was wrong at least.
