Reboots/resets randomly - RRF 3.5.0-b4

dc42

@Exerqtor was the machine idle or printing?

Exerqtor

@dc42
Idle. It's been a couple more crashes since last post to, also while idle.

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 19 of 40 (28 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 124180 of which 12 recycled
Never used RAM 12012, free system stack 186 words
Tasks: NETWORK(2,nWait,18.4%,219) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,662) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,delaying,1.2%,108) MAIN(1,running,78.5%,704) IDLE(0,ready,0.9%,29) AIN(4,delaying,0.8%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:18:41 ago, cause: software
Last software reset at 2023-07-25 20:51, reason: HardFault invState, Gcodes spinning, available RAM 11236, slot 1
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001b0 00000002 200014ec 00000000 ffffffff 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 20032174 00000800 20035e68 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 b5ddc1d1
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1121100, completed 1121100, timed out 0, errs 0
MCU temperature: min 36.3, current 36.9, max 41.8
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/29, heap memory allocated/used/recyclable 2048/1692/1296, gc cycles 137
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 174, read errors 0, write errors 1, ifcnt 103, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 16, read errors 0, write errors 1, ifcnt 103, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 111, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 110, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 108, reads 36347, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 175, reads 36350, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 175, reads 36350, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-07-25 21:10:08
Cache data hit count 1876944189
Slowest loop: 8.00ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.3ms, write time 2.2ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P100                                                                       " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 10098, received 22458, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 261, free buffers 18 (min 17), ts 5606/5605/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 10.73ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 43124
WiFi IP address 192.168.10.50
Signal strength -52dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

dc42

@Exerqtor can you try un-configuring some devices to see whether that stops the board from crashing when it is sitting idle. For example, you could un-configure everything on the expansion boards; or un-configure all temperature sensors.

Exerqtor

@dc42
Sure, i'm up for trying. What would be the best approach to do this in your opinion (i'm not sure how to "un-configure" it once it's been configured

And would you suggest putting it in daemon or just make it a custom config.g to be ran as a test with mentioned stuff not configured and let the machine stay idle to see if it mitigates the crashing?

dc42

@Exerqtor I suggest temporarily commenting out the corresponding lines in config.g and rebooting.

Exerqtor

TLDR: Can a bad / "EOL" SD-card have been at fault for the issues I've been having?

@dc42
Never got around to change the lines since the printer was running stable on 3.5b2 and i had some prints going.

I went through with buying a genuine Mini 5+ instead since i was/am in the impression that issue was most likely HW related. It still haven't arrived so I cant confirm that or not.

BUT on sunday evening while doing some routine maintenance (printer powered down, and power cord pulled) something weird happened when I finished up and booted the printer. It started all normally and i went through to home it but then PD promted me that i was missing all the homing files (one after one once i tried axis by axis), i went forrward to try turning on the chamber lights with it's macro: same issue, it's reported missing as well.

Baffeled i tried heating up the hotend to 75c just to see if the hotend fan turned on or not, it did. Turned off the heater and let it settle to where the fan turned off again. Started to think something must have gone wrong in the bootup (or something), so i powet toggeled the printer, this time PD warned about some JSON error while booting then just said "connecting" in top right corner and everything was unresponsive, couldn't connect to DWC either.

That's when i started to suspect something was going on with the SD card, powered down again and pulled the card out and put it in my computer, and sure AF it was corrupted with only a handfull of files readable, and ALL the files in sys/ and subfolders had changed names and exentions to something totally nonsense and went corrupted despite changing them to the right file extension.

Tried running some recovery software on it, and managed to pull everything from filaments/, firmware/ & most of the job files in gcode/ except one file. So yesterday i bought a new namebrand class 10 sdcard and started rebuilding the printer setup on the new card. And i just managed to boot and run the checks before going to bed.

With everything confirmed working it started to dawn on me that maybe a shoddy SD card might have been the culprit for this reboot issue to, so I got back up from bed and flashed 3.5b4 to both the MB and Toolboard and ran a new set of initial checks to see that everything was fine (which it was) and went back to bed.

Checked the printer now after the morning walk with the dog, and still bo reboots/crashes/resets.
SOOOO can a shoddy SD card have caused these issues all along? Or is the corrupted card a totally standalone problem?

dc42

@Exerqtor the only way I can think of that a corrupt SD card could cause the problem would be if RRF read a corrupt command from it that caused RRF to overflow an array bound, or something like that.

All the code I added to RRF does check that commands don't cause array bounds to be exceeded, but there might be some older code that doesn't.

Exerqtor

@dc42 Ok, i to doubt the SD card was the culprit but i thought i should air the theory since it happened.

Still running 3.5b4 in idle at home, so i'm curious if it's gotten any new resets by the time i finish work 🤷‍♂️

Curious to see if the behaviour changes with the new board to ofc.

Exerqtor

Just had a reset now, so that comfirms it not being the SD-card lol.

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 1 of 40 (28 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 125644 of which 12 recycled
Never used RAM 10548, free system stack 182 words
Tasks: NETWORK(1,ready,17.1%,203) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,1.2%,108) MAIN(1,running,80.6%,704) IDLE(0,ready,0.2%,29) AIN(4,delaying,0.8%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:44:41 ago, cause: software
Last software reset at 2023-08-15 18:10, reason: HardFault invState, Gcodes spinning, available RAM 9052, slot 0
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001af 00000002 200014ec 00000000 2003338b 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 2003095c 00000800 20035e98 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 b5dd60fa
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 2681568, completed 2681568, timed out 0, errs 0
MCU temperature: min 36.1, current 36.9, max 42.2
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/564/128, gc cycles 125
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 126, reads 47121, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 14, read errors 0, write errors 1, ifcnt 126, reads 47121, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 98, reads 47121, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 99, reads 47120, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 99, reads 47121, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 47124, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 47124, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-08-15 18:55:09
Cache data hit count 4294967295
Slowest loop: 13.48ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 5.3ms, write time 4.5ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1                                                                        " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 24140, received 53662, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 316, free buffers 18 (min 17), ts 13407/13406/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 6.07ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42972
WiFi IP address 192.168.10.50
Signal strength -57dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Hopefully the new board arrives tomorrow or tuesday, let's hope that solves it.

Nuramori

I had a reset as well, this one ANOTHER out of memory fault tied to the tool-board/CAN bus. It's at the point I can't do any large prints without the machine resetting half way through and wasting filament. Incredibly frustrating when you have it running 1-2 days only to quit on you before it finishes.

dc42

@Nuramori please try 3.5.0-rc.1.

Exerqtor

@dc42 said in Reboots/resets randomly - RRF 3.5.0-beta4:

@Nuramori please try 3.5.0-rc.1.

Sorry for the lack of engagement but I haven't had time to fiddle with the printer in a while. I've been running 3.5.0-rc.1 since i got the new board. And it's still rebooting/crashing from time to time, but it's not been as often.

Here is the last few occations from the debug logg and a fresh M122 i just ran.

power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-09 01:20:53 [warn] Date and time set at power up + 00:00:10
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-14 16:46:21 [warn] Date and time set at power up + 00:24:02
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-15 04:22:33 [warn] Date and time set at power up + 00:00:08
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-15 07:16:09 [warn] Date and time set at power up + 00:00:08
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-21 06:57:47 [warn] Date and time set at power up + 00:00:08
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-21 11:21:14 [warn] Date and time set at power up + 00:00:13

M122:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1 (2023-08-31 16:16:56) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102836
Dynamic ram: 125300 of which 0 recycled
Never used RAM 9800, free system stack 135 words
Tasks: NETWORK(2,nWait,17.6%,233) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,243) CanReceiv(6,nWait,0.0%,773) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,0.7%,74) MAIN(1,running,80.5%,704) IDLE(0,ready,0.3%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:47:14 ago, cause: software
Last software reset at 2023-10-21 11:20, reason: HardFault invState, Gcodes spinning, available RAM 13184, slot 2
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001af 00000002 200014ec 00000000 20032bfb 0009df2d 00000000 600f0000 00000000 00000000 00000000 00000000 20031a1c 00000800 20035710 2002bf30 20018668 2002bd9d 20018668 2001e880 0002fedf 00000000 00000000 00000000 20012058 00000014 0893d091
Error status: 0x04
Aux0 errors 0,1,0
MCU revision 3, ADC conversions started 2835341, completed 2835340, timed out 0, errs 0
MCU temperature: min 34.4, current 45.3, max 45.8
Supply voltage: min 22.4, current 24.0, max 26.6, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/34, heap memory allocated/used/recyclable 2048/964/508, gc cycles 134
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 109, reads 18073, writes 43, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 109, reads 18073, writes 43, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 86, reads 18091, writes 25, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 85, reads 18091, writes 24, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 86, reads 18091, writes 25, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-10-21 12:08:16
Cache data hit count 4294967295
Slowest loop: 29.65ms; fastest: 0.12ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.3ms, write time 28.1ms, max retries 0
=== Move ===
DMs created 83, segments created 26, maxWait 1519826ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 20, on retry 8, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 47, completed 47, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 11], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.4
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000807
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 25544, received 58175, lost 0, boc 0
Longest wait 3ms for reply type 6013, peak Tx sync delay 258, free buffers 26 (min 25), ts 14175/14174/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 46.60ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(2) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 3 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.37, flash size 2097152, free heap 37884
WiFi IP address 192.168.10.50
Signal strength -50dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Changing the thread title to reflect on the issue persisting to rc1

Leonard03

Hello! I started to have the same symptoms with my board (https://forum.duet3d.com/topic/33857/duet-2-wifi-hardfault-invstate) and I wonder if this tyoe of reset has something to do with the powersupply itself. Can a quick drop in the powersupply output result in this type of crash? A drop that is long enogh to cause a drop in the 3v3 circuit of the processor?
Just an idea..

NeoDue

Just for the sake of completeness (the other two issues I posted here are more pressing to me) - I have those reboots as well, running 3.5.0rc1 on a Duet 3 6HC with a Duet 3 wifi module on the latest firmware. In addition to that, I sometimes see connection losses of both the webinterface and the PanelDue that can only be cured by a reset.

Thankfully they did not happen during a print as of now, but rather when the printer has been sitting idle for an extended period of time. That might be caused by the fact that it usually takes the printer about 10+ hours to reset as far as I have noticed by now - all prints I did so far were well finished by then.

When this happens again, I will add an M122 printout here. So far,, the only thing I noticed on previous cases was the recurring note "gcodes spinning" as reset reason in all those cases.

Exerqtor

"Glad" to see i'm not the only one having these crashes, must admit that.

I had another crash mid print roughly an hour ago

debug log:

power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-21 22:24:09 [warn] Date and time set at power up + 00:00:09

M122:

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1 (2023-08-31 16:16:56) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102836
Dynamic ram: 124928 of which 0 recycled
Never used RAM 10436, free system stack 136 words
Tasks: NETWORK(1,ready,108.3%,228) HEAT(3,nWait,0.2%,335) Move(4,nWait,0.0%,261) CanReceiv(6,nWait,0.3%,773) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,342) TMC(4,nWait,4.3%,74) MAIN(1,running,529.8%,704) IDLE(0,ready,1.7%,29) AIN(4,delaying,5.3%,264), total 649.8%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 01:52:48 ago, cause: software
Last software reset at 2023-10-21 22:24, reason: HardFault invState, Gcodes spinning, available RAM 4720, slot 0
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001ae 00000002 200014ec 00000000 20032bfa 0009df2d 00000000 600f0000 00000000 00000000 00000000 00000000 200301d4 00000800 20035710 2002bf00 20018668 2002bd9d 20018668 2001e880 0002fedf 00000000 00000000 00000000 20012058 00000014 b5dd8a35
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 6768325, completed 6768325, timed out 0, errs 0
MCU temperature: min 37.4, current 41.5, max 48.6
Supply voltage: min 2.2, current 24.1, max 25.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/34, heap memory allocated/used/recyclable 2048/1572/1116, gc cycles 318
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 35, reads 28491, writes 35, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 35, reads 28491, writes 35, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 20, reads 28506, writes 20, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 19, reads 28506, writes 19, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 19, reads 28507, writes 19, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-10-22 00:16:47
Cache data hit count 4294967295
Slowest loop: 13.37ms; fastest: 0.12ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.0ms, write time 4.6ms, max retries 0
=== Move ===
DMs created 83, segments created 15, maxWait 5875833ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 7, on retry 6, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 25, completed 25, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000807
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 60937, received 138751, lost 0, boc 0
Longest wait 2ms for reply type 6034, peak Tx sync delay 1839, free buffers 26 (min 25), ts 33842/33840/0
Tx timeouts 0,0,1,0,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 8.41ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42952
WiFi IP address 192.168.10.50
Signal strength -51dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Exerqtor

And another crash mid print, this is starting to become a REALLY annoying problem!

power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-22 10:21:54 [warn] Date and time set at power up + 00:00:08

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1 (2023-08-31 16:16:56) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 3 of 40 (32 max)
=== RTOS ===
Static ram: 102836
Dynamic ram: 124104 of which 0 recycled
Never used RAM 11620, free system stack 186 words
Tasks: NETWORK(1,ready,17.3%,213) HEAT(3,nWait,0.0%,335) Move(4,nWait,0.0%,352) CanReceiv(6,nWait,0.0%,796) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,342) TMC(4,nWait,0.7%,108) MAIN(1,running,80.9%,704) IDLE(0,ready,0.3%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:20:17 ago, cause: software
Last software reset at 2023-10-22 10:21, reason: HardFault invState, Expansion spinning, available RAM 7500, slot 1
Software reset code 0x4072 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001ae 00000002 200014ec 00000000 20032c0a 0009df2d 00000000 600f0000 00000000 00000000 00000000 00000000 20031a2c 00000800 2002bf00 2002bf00 00000001 2002bd9d 20018668 2001e880 0002fedf 00000000 00000000 00000000 20012058 00000014 b5dd8a35
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1217157, completed 1217157, timed out 0, errs 0
MCU temperature: min 41.0, current 41.7, max 48.4
Supply voltage: min 2.2, current 24.1, max 25.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/2008/1572, gc cycles 56
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64030, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64030, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-10-22 10:42:02
Cache data hit count 2002338155
Slowest loop: 16.49ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.1ms, write time 4.4ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 10966, received 24933, lost 0, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 736, free buffers 26 (min 25), ts 6086/6084/0
Tx timeouts 0,0,1,0,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 24.59ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 43128
WiFi IP address 192.168.10.50
Signal strength -46dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Needless to say that the hypothesis of the core issue being a non Duet board was wrong at least.

dc42

@Exerqtor this thread is very long so I haven't time to re-read every message although I did search for relevant words, with no hits. So apologies if we have covered this before. However:

'invState' hard faults typically show nonsensical stack data, different every time. We suspect that these faults are caused by transients or ESD. Have you grounded your hot end and extruder metalwork to Duet ground, either directly or through a resistor? This will avoid the act of extruding filament causing static charge to build up, which is likely to flash cover to the thermistor connections.

Exerqtor

@dc42 Yeah it's getting pretty long, but starting a new/follow-up thread would potentially "spread" info so I guess keeping everything here still is the best way to go(?).

Uuhm I haven't done so, I'm running a LGX-lite and Revo-micro so there aren't really any obvious places for me to ground either of them.

If you have any ideas of good ways to ground those I'm all ears and will try to deploy it ASAP 😅

If ESD turns out being the issue, it's quite weird that it manifests after FW update though?

Leonard03

@Exerqtor here I cand give you an ideea about grounding. After I had almost the same issue about one year ago (?) as David said, I've grounded all seven steppers on by printer by a ring connector (if this is how is called) and three washers between sthe motors and the mounting points. The washers should be the same thickness as the ring of the connector. Not the most elegant way, but it does the job. All connections I've done to the ATX PSU and from there on is groundes by the power plug (seems that the duet2wifi is grounded with the PSU by the negative terminal).

Abount grounding the hotend, I'm running an E3D V6 gold edition with the PT100 and the DB but back then I've contacted E3D but they sayd that the heatsink of the hotend is not conductive, so the only grounding for the filament path is by the motor shaft being in contact with it

Exerqtor

@Leonard03 Thanks for weighing in, but i'm pretty sure this is the oposite in my case. Since the extruder stepper ain't in contact with the filament in any way. And the extruder motor aren't in contact with any metal in a regular LGX lite either. Everything on a Revo micro however is conductive

Also, everything is mounted in/on a Voron Stealthburner (printed ABS parts) , no metal conducting anything betwen the extruder and hotend that way either.