Reboots/resets randomly - RRF 3.5.0-b4

dc42

@Exerqtor I'm wondering whether it could possibly be a hardware issue, because I haven't been able to reproduce it and nobody else has reported it. Which country are you in and which Duet3D reseller do you use?

Exerqtor

@dc42
Might be the case, this perticular board ain't a original duet3 board but a Fysetc "Big Dipper" (killed the processor on my Duet 2 so I needed a subsitute board ASAP, and couldn't justify cashing out for a original duet 3 mini & addon board to the mrs 🤦‍♂️).

It haven't had a single crash since the last one i reportet two weeks ago though. Even with the extra load we added.

Just started a 2.3hr print on it now to see if it wil act up again.

droftarts

@Exerqtor Please be up-front about using clone Duet boards. It would save a lot of time supporting, and trying to resolve issues with, boards who's quality are beyond the scope of what Duet3D can control.

Regarding the Fysetc Big Dipper board, when the TeamGloomy devs took a look at it, they weren't that impressed. When looking at the schematics, it doesn't include as much filtering, and some other corners have been cut, compared to the Duet 3D Mini 5+.

Ian

dc42

@Exerqtor as I haven't been able to reproduce it, and nobody else has reported it, and it's a clone board, I'll park this issue unless/until someone else reports a similar issue.

The SAME54P20A processor we use in the Duet 3 Mini has been almost impossible to purchase for the last two years, so I wouldn't rule out the possibility that the clone boards use reject/remarked chips.

Exerqtor

@droftarts It wasn't like I was actively trying to withold any info in that regard. Since it was a non-issue before the specific FW update I didn't even think about it possibly being and HW issue given the circumstances.

I'm in no way doubtfull of the original duets being of way superior quality all across the board, but the circumstances simply didn't allow for it when I had to replace the board i killed.

@dc42 Yeah I haven't had any more issues myself either (knock on wood) so I have no idea what was happening.

And just to be clear, I didn't mean or try to gasslight anyone with not specifying it being a clone board since I assumed it to "simply" being a FW issue introduced to the beta in question.

Exerqtor

@Exerqtor
Just checked the printer and saw it had crashed again, about 30min after the job from yesterday was finished.

Heres the M122 in case it might shine some new light on it 🤷‍♂️

5.7.2023, 18:19:11	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 1 of 40 (28 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 125612 of which 12 recycled
Never used RAM 10580, free system stack 180 words
Tasks: NETWORK(1,ready,1261.8%,211) HEAT(3,nWait,9.0%,327) Move(4,nWait,0.9%,344) CanReceiv(6,nWait,17.5%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,2.8%,351) TMC(4,delaying,423.1%,108) MAIN(1,running,2878.0%,704) IDLE(0,ready,1126.3%,29) AIN(4,delaying,293.3%,266), total 6012.7%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 19:08:33 ago, cause: software
Last software reset at 2023-07-04 23:10, reason: HardFault invState, Gcodes spinning, available RAM 2364, slot 0
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001ae 00000002 200014ec 00000000 2003338a 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 200321a4 00000800 20035e98 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 b5ddc1d1
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 68913225, completed 68913223, timed out 0, errs 0
MCU temperature: min 34.5, current 35.5, max 41.6
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/32, heap memory allocated/used/recyclable 2048/796/372, gc cycles 21269
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 194, reads 38882, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 194, reads 38882, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 93, reads 38882, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 93, reads 38882, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 92, reads 38882, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 21, reads 38885, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 21, reads 38885, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-07-05 18:19:11
Cache data hit count 4294967295
Slowest loop: 37.47ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.4ms, write time 3.7ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P100                                                                       " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 620226, received 1378327, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 273, free buffers 18 (min 17), ts 344566/344565/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 22.25ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42780
WiFi IP address 192.168.10.50
Signal strength -48dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

dc42

@Exerqtor that's a different crash from the previous ones but probably in the same area. Looks like a similar cause: a few stack words have been cleared to zeros during a DMA transfer between the WiFi module and the main processor.

The only thing I can suggest (and it's a long shot) is to try reducing the SPI clock speed of transfers between the WiFi module and the main processor. You can do this by sending:

M589 L{0x3002}

M122 should then display the WiFi clock register as 00003002.

Exerqtor

@dc42
Interesting 😶

I'll try that later, is guess this something that needs to be done on every boot right (so I'll put it in config.g)?

dc42

@Exerqtor it won't work in config.g because the WiFi interface isn't enabled until config.g completes.

Exerqtor

@dc42 noted

droftarts

@Exerqtor if it is WiFi related as @dc42 suggests, I note you’re on WiFi firmware 2.2beta4 . Check you have the later version of this, as I think there was an issue with the early release of this. It might also be worth reverting to 1.27, in the RRF 3.4.5 release.

Ian

Exerqtor

@droftarts I'm pretty sure i installed the newest one once David found and patched the issue, but ofc. I can download and install the newest one avalible on git to make sure.

I'm trying to avoid rolling back, if possible (since i would much rather try to have it fixed). If my memory serves right I had no issues on 3.5b2 either, and these issues was introduced with beta 3.

Otherwise, i did a total of 12hours printing yesterday almost back to back without any issues. The printer however totally lost it's shit during the day today while idle 🤣

2023-07-09 23:07:03 [warn] Finished printing file 0:/gcodes/Bottom_0.2mm_ABS_0.4n_5h46m.gcode, print time was 5h 37m
2023-07-09 23:07:08 [info] G10 P0 R0 S0                                                                   
2023-07-09 23:07:08 [debug] ABS filament unloaded & pressure advance disabled
2023-07-09 23:07:08 [debug] ABS filament unloaded & pressure advance disabled
2023-07-10 06:30:48 [warn] HTTP client 192.168.10.100 login succeeded (session key 1350785786)
2023-07-10 11:21:16 [warn] HTTP client 192.168.10.100 login succeeded (session key 3991901514)
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.4 (2023-06-08 23:40:14)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:08 [warn] HTTP client 192.168.10.100 login succeeded (session key 572637009)
2023-07-10 12:25:29 [warn] Date and time set at power up + 00:00:08
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.4 (2023-06-08 23:40:14)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:08 [warn] HTTP client 192.168.10.100 login succeeded (session key 927937073)
2023-07-10 12:37:30 [warn] Date and time set at power up + 00:00:08
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.4 (2023-06-08 23:40:14)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:08 [warn] HTTP client 192.168.10.100 login succeeded (session key 1498099025)
2023-07-10 15:57:31 [warn] Date and time set at power up + 00:00:08
2023-07-10 17:13:09 [warn] HTTP client 192.168.10.226 login succeeded (session key 4209983836)

10.7.2023, 17:22:44	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 14 of 40 (39 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 124180 of which 12 recycled
Never used RAM 12012, free system stack 180 words
Tasks: NETWORK(2,nWait,20.8%,219) HEAT(3,nWait,0.0%,327) Move(4,nWait,0.0%,344) CanReceiv(6,nWait,0.0%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,1.2%,108) MAIN(1,running,73.9%,704) IDLE(0,ready,3.2%,29) AIN(4,delaying,0.8%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 01:25:22 ago, cause: software
Last software reset at 2023-07-10 15:57, reason: HardFault invState, Gcodes spinning, available RAM 11988, slot 0
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001ae 00000002 200014ec 00000000 2003335a 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 20031144 00000800 20035e68 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 02005448
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 5122213, completed 5122213, timed out 0, errs 0
MCU temperature: min 35.3, current 35.9, max 39.0
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/28, heap memory allocated/used/recyclable 2048/916/532, gc cycles 1519
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 35, reads 6867, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 35, reads 6867, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 155, reads 6867, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 155, reads 6867, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 154, reads 6867, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 6869, writes 10, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x6a
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 6870, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-07-10 17:22:44
Cache data hit count 4294967295
Slowest loop: 22.50ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.6ms, write time 3.6ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P100                                                                       " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 46107, received 102482, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 322, free buffers 18 (min 17), ts 25611/25610/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 91.05ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42796
WiFi IP address 192.168.10.50
Signal strength -53dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

The fist of those crashes was with reduced SPI clock speeds, but the subsequent ones should be with default speed.

EDIT:

Just re-downloaded and installed the WifiServer from Release 3.5.0-beta.4 again. Lets see if it's any changes, I'm however 99,9% sure i allready had the latest release of it installed.

Exerqtor

That reinstall of the 2.1b4 WiFiServer FW didin't help, just had another crash while swapping out filaments before starting a print.

11.7.2023, 11:10:03	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 15 of 40 (33 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 125612 of which 12 recycled
Never used RAM 10580, free system stack 188 words
Tasks: NETWORK(2,nWait,20.3%,213) HEAT(3,nWait,0.0%,327) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,1.2%,108) MAIN(1,running,74.0%,704) IDLE(0,ready,3.7%,29) AIN(4,delaying,0.8%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:02:00 ago, cause: software
Last software reset at 2023-07-11 11:07, reason: HardFault invState, Platform spinning, available RAM 9188, slot 1
Software reset code 0x4060 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001af 00000002 200014ec 00000000 2003335b 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 2003092c 00000800 20035e68 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 02005448
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 120690, completed 120690, timed out 0, errs 0
MCU temperature: min 40.5, current 40.7, max 43.8
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/32, heap memory allocated/used/recyclable 2048/456/32, gc cycles 36
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 66, reads 10933, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 66, reads 10932, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 173, reads 10932, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 174, reads 10932, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 173, reads 10933, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 65, reads 10936, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 65, reads 10935, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-07-11 11:10:01
Cache data hit count 202352397
Slowest loop: 37.82ms; fastest: 0.14ms
=== Storage ===
Free file entries: 17
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.2ms, write time 4.5ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 1094, received 2447, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 328, free buffers 18 (min 17), ts 604/603/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 22.06ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 39884
WiFi IP address 192.168.10.50
Signal strength -60dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Rolling back the WiFiServer FW to 1.27 now to see if that helps.

Exerqtor

Rolling back to WiFiServer 1.27 didn't help either.

2023-07-11 13:15:53 [warn] Finished printing file 0:/gcodes/A3-divided-raised_0.2mm_ABS_0.4n_1h55m.gcode, print time was 1h 55m
2023-07-11 13:15:58 [info] G10 P0 R0 S0                                                                   
2023-07-11 13:15:58 [debug] ABS filament unloaded & pressure advance disabled
2023-07-11 13:15:58 [debug] ABS filament unloaded & pressure advance disabled
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.4 (2023-06-08 23:40:14)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:03 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:09 [warn] HTTP client 192.168.10.100 login succeeded (session key 3300568348)
2023-07-11 14:21:10 [warn] Date and time set at power up + 00:00:09
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.4 (2023-06-08 23:40:14)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:03 [warn] WiFi module started
power up + 00:00:09 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:09 [warn] HTTP client 192.168.10.100 login succeeded (session key 1905162507)
2023-07-11 17:44:06 [warn] Date and time set at power up + 00:00:09
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.4 (2023-06-08 23:40:14)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:03 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:09 [warn] HTTP client 192.168.10.100 login succeeded (session key 1141699597)
2023-07-11 18:32:59 [warn] Date and time set at power up + 00:00:09

11.7.2023, 18:57:10	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 124216 of which 0 recycled
Never used RAM 11988, free system stack 186 words
Tasks: NETWORK(1,ready,21.0%,219) HEAT(3,nWait,0.0%,327) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,delaying,1.2%,108) MAIN(1,running,73.1%,704) IDLE(0,ready,3.8%,29) AIN(4,delaying,0.8%,266), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:24:18 ago, cause: software
Last software reset at 2023-07-11 18:32, reason: HardFault invState, Platform spinning, available RAM 11980, slot 1
Software reset code 0x4060 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001ae 00000002 200014ec 00000000 2003335a 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 2003217c 00000800 20035e70 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 b5ddc1d1
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1458383, completed 1458383, timed out 0, errs 0
MCU temperature: min 35.3, current 35.9, max 39.0
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/28, heap memory allocated/used/recyclable 2048/468/84, gc cycles 428
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 143, reads 1472, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 143, reads 1472, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 229, reads 1472, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 231, reads 1472, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 230, reads 1472, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 98, reads 1475, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 98, reads 1475, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-07-11 18:57:08
Cache data hit count 2422773404
Slowest loop: 22.63ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.7ms, write time 12.4ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P100                                                                       " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 13133, received 29207, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 223, free buffers 18 (min 17), ts 7292/7291/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 22.76ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 1.27
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 25544
WiFi IP address 192.168.10.50
Signal strength -57dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

EDIT 12/7-23:

With the crashes being just as bad now as with the 2.1b4 WiFiServer FW i'm rolling everything back to 3.5b2 since that's the last "stable" version I can remember being on in regards to the crashing. If the crashing stops there I would assume it's safe to say it's a change post 3.5b2 that's causing the behaviour.

One last M122 from 3.5b4:

12.7.2023, 11:43:13	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 1 of 40 (30 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 125612 of which 12 recycled
Never used RAM 10580, free system stack 182 words
Tasks: NETWORK(1,ready,17.2%,203) HEAT(3,nWait,0.0%,341) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,1.2%,108) MAIN(1,running,78.3%,704) IDLE(0,ready,2.3%,29) AIN(4,delaying,0.8%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 01:22:08 ago, cause: software
Last software reset at 2023-07-12 10:21, reason: HardFault invState, Expansion spinning, available RAM 9476, slot 2
Software reset code 0x4072 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001ae 00000002 200014ec 00000000 2003336a 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 2003115c 00000800 20035e80 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 b5ddc1d1
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 4929142, completed 4929140, timed out 0, errs 0
MCU temperature: min 35.9, current 36.7, max 42.1
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/32, heap memory allocated/used/recyclable 2048/1404/980, gc cycles 781
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 1, ifcnt 17, reads 54838, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 17, reads 54838, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 80, reads 54839, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 82, reads 54839, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 82, reads 54839, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 175, reads 54842, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 175, reads 54842, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-07-12 11:43:10
Cache data hit count 4294967295
Slowest loop: 37.38ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.7ms, write time 3.9ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P100                                                                       " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 44368, received 98617, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 270, free buffers 18 (min 17), ts 24645/24644/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 22.05ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 1.27
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 22464
WiFi IP address 192.168.10.50
Signal strength -59dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

Exerqtor

@dc42
I've been running the printer more or less nonstop (except during the night) since the 12th on the full 3.5b2 release without a single crash.

So it seems to me it's some change that got implemented post 3.5b2 that makes my machine act up at least.

EDIT 16th July:

Another two days without any issues running 3.5 beta2.

Exerqtor

@dc42
Still not a single crash with 3.5b2, so i'll go out on a limb and claim that it's a 3.5b3/b4 issue and not HW.

dc42

@Exerqtor can you still get it to crash when you run 3.5.0-beta.4 ?

It could still be a hardware issue, because the memory layout changes between firmware versions. If it's a hardware issue then it might be benign in older firmware versions, e.g. if memory still gets corrupted but that memory is an unused area of the stack instead of a recently-used area.

If it was a a firmware issue, then I would expect to be able to reproduce the reset on a genuine Duet using your build configuration; but I ran it twice for 8 hours at a time and it didn't crash. I would also expect other users to report it.

Exerqtor

@dc42
Gotcha! I'll update to b4 again tomorrow and see how it plays out. If it's still an issue i simply have to run b2 until I get a genuine Duet 3 Mini.

Exerqtor

Updated to 3.5b4 yesterday evening(fresh binaries from github) , and the crashing started again.

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 1 of 40 (38 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 124204 of which 12 recycled
Never used RAM 11988, free system stack 182 words
Tasks: NETWORK(1,ready,140.3%,219) HEAT(3,nWait,0.2%,327) Move(4,nWait,0.0%,348) CanReceiv(6,nWait,0.4%,672) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.1%,351) TMC(4,nWait,8.9%,108) MAIN(1,running,260.0%,704) IDLE(0,ready,6.7%,29) AIN(4,delaying,6.1%,266), total 422.7%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 03:40:27 ago, cause: software
Last software reset at 2023-07-25 06:15, reason: HardFault invState, Platform spinning, available RAM 11988, slot 1
Software reset code 0x4060 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001b0 00000002 200014ec 00000000 ffffffff 0009d43d 00000000 600f0000 00000000 00000000 00000000 00000000 20032174 00000800 20035e68 2002c558 20018670 2002c3f5 20018670 2001e920 0002fe6f 00000000 00000000 00000000 20012058 00000014 00000000
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 13227701, completed 13227701, timed out 0, errs 0
MCU temperature: min 35.5, current 36.1, max 39.4
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/29, heap memory allocated/used/recyclable 2048/1884/1488, gc cycles 1629
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 216, reads 22838, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 216, reads 22837, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 234, reads 22837, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 234, reads 22837, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 232, reads 22838, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 76, reads 22839, writes 10, timeouts 2, DMA errors 0, CC errors 0, failedOp 0x41
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 76, reads 22840, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-07-25 09:55:34
Cache data hit count 4294967295
Slowest loop: 24.26ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.5ms, write time 4.5ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P100                                                                       " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 119057, received 264597, lost 0, boc 0
Longest wait 4ms for reply type 6053, peak Tx sync delay 274, free buffers 18 (min 17), ts 66139/66138/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 22.45ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42968
WiFi IP address 192.168.10.50
Signal strength -54dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

dc42

@Exerqtor was the machine idle or printing?