Reboots/crashes - RRF ≤3.5.0-rc1

[Exerqtor]

Here we go:

debuglog:

2023-11-16 03:59:44 [warn] HTTP client 192.168.10.xx login succeeded (session key 1771356456)
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1+ (2023-11-15 20:11:19)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1+ (2023-11-15 20:11:19)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:04 [warn] Error: Heater 1 fault: failed to read sensor: notReady
power up + 00:00:04 [info] M291: - Event notification - Heater 1 fault: failed to read sensor: notReady
power up + 00:00:04 [debug] - Event notification -
power up + 00:00:04 [debug] Heater 1 fault: failed to read sensor: notReady
power up + 00:00:06 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:06 [warn] WiFi module is idle
power up + 00:00:08 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:08 [warn] WiFi module is idle
power up + 00:00:11 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:11 [warn] WiFi module is idle
power up + 00:00:13 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:13 [warn] WiFi module is idle
power up + 00:00:15 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:15 [warn] WiFi module is idle
power up + 00:00:18 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:18 [warn] WiFi module is idle
power up + 00:00:20 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:20 [warn] WiFi module is idle
power up + 00:00:22 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:22 [warn] WiFi module is idle
power up + 00:00:25 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:25 [warn] WiFi module is idle
power up + 00:00:27 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:27 [warn] WiFi module is idle
power up + 00:00:30 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:30 [warn] WiFi module is idle
power up + 00:00:32 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:32 [warn] WiFi module is idle
power up + 00:00:34 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:34 [warn] WiFi module is idle
power up + 00:00:37 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:37 [warn] WiFi module is idle
power up + 00:00:41 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.x
power up + 00:00:42 [warn] HTTP client 192.168.10.xx login succeeded (session key 1408720838)
2023-11-16 09:15:05 [warn] Date and time set at power up + 00:00:42

M122:

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1+ (2023-11-15 20:11:19) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (29 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 123820 of which 0 recycled
Never used RAM 11896, free system stack 186 words
Tasks: NETWORK(1,ready,3212.5%,232) HEAT(3,nWait,4.5%,327) Move(4,nWait,0.5%,352) CanReceiv(6,nWait,9.0%,774) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,1.4%,350) TMC(4,nWait,129.2%,108) MAIN(1,running,1199.1%,670) IDLE(0,ready,45.8%,29) AIN(4,delaying,153.3%,264), total 4755.3%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 06:23:49 ago, cause: software
Last software reset at 2023-11-16 09:14, reason: AssertionFailed, Gcodes spinning, available RAM 10964, slot 2
Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00487000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000904 000af288 0002ddbf 2002bed4 2002be01 000001a8 2002c100 20031c40 2002c118 2001e888 00000000 a5a5a5a5 00000000 00000000 00000000 00000000 20031c4c 00000800 20035940 2002c100 20018668 2002bf9d 20018668 2001e888 00030037 00000000 00000000
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 23029641, completed 23029641, timed out 0, errs 0
MCU temperature: min 34.7, current 35.1, max 37.8
Supply voltage: min 23.0, current 24.1, max 26.0, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/716/280, gc cycles 1071
Events: 1 queued, 1 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 1, ifcnt 170, reads 32406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 222, read errors 0, write errors 1, ifcnt 170, reads 32406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 127, reads 32406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 125, reads 32406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 126, reads 32406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-11-16 15:38:13
Cache data hit count 4294967295
Slowest loop: 12.92ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.2ms, write time 4.1ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 207276, received 472154, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6057, peak Tx sync delay 275, free buffers 26 (min 25), ts 115148/115147/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 5.98ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42960
WiFi IP address 192.168.10.x
Signal strength -52dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

[dc42]

@Exerqtor thanks, that's just what I needed to pinpoint what's going on. A DMA transfer is overwriting the return address on the stack. Unless I can spot the reason, I'll need to add some more debug to the firmware and ask you to run in again to find out why.

[Exerqtor]

@dc42 Thats great news! Starting to get the feeling it might be some light breaking through the clouds now 🤣

Just put in what ever you might need and i'll give it a go, can't really use the printer in this state anyways lol.

[dc42]

@Exerqtor thanks, I've had to work on other things today but I'll come back to this very soon. I'm still puzzled that it's not reported more widely.

[Exerqtor]

@dc42 No problem! Yeah it's struck me as odd too 🤷‍♂️

[Exerqtor]

@dc42 New crash:
debug:

2023-11-17 21:01:32 [warn] HTTP client 192.168.10.xx login succeeded (session key 3428338341)
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1+ (2023-11-15 20:11:19)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1+ (2023-11-15 20:11:19)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] Error: Heater 1 fault: failed to read sensor: notReady
power up + 00:00:04 [info] M291: - Event notification - Heater 1 fault: failed to read sensor: notReady
power up + 00:00:04 [debug] - Event notification -
power up + 00:00:04 [debug] Heater 1 fault: failed to read sensor: notReady
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:06 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:06 [warn] WiFi module is idle
power up + 00:00:08 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:08 [warn] WiFi module is idle
power up + 00:00:11 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:11 [warn] WiFi module is idle
power up + 00:00:13 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:13 [warn] WiFi module is idle
power up + 00:00:16 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:16 [warn] WiFi module is idle
power up + 00:00:18 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:18 [warn] WiFi module is idle
power up + 00:00:20 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:20 [warn] WiFi module is idle
power up + 00:00:23 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:23 [warn] WiFi module is idle
power up + 00:00:25 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:25 [warn] WiFi module is idle
power up + 00:00:27 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:27 [warn] WiFi module is idle
power up + 00:00:30 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:30 [warn] WiFi module is idle
power up + 00:00:32 [warn] Error: WiFi module reported: Failed to load credentials
power up + 00:00:32 [warn] WiFi module is idle
power up + 00:00:36 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.x
power up + 00:00:36 [warn] HTTP client 192.168.10.xx login succeeded (session key 2751057796)
2023-11-18 07:01:07 [warn] Date and time set at power up + 00:00:36

M122:

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1+ (2023-11-15 20:11:19) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (34 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 123844 of which 0 recycled
Never used RAM 11872, free system stack 186 words
Tasks: NETWORK(1,ready,16.7%,219) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,0.7%,108) MAIN(1,running,81.1%,670) IDLE(0,ready,0.6%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:22:02 ago, cause: software
Last software reset at 2023-11-18 07:00, reason: AssertionFailed, Gcodes spinning, available RAM 11268, slot 1
Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000904 000af288 0002ddbf 2002bed4 2002be01 000001a8 2002c100 200303f8 2002c118 2001e888 00000000 a5a5a5a5 00000000 00000000 00000000 00000000 20030404 00000800 20035940 2002c100 20018668 2002bf9d 20018668 2001e888 00030037 00000000 00000000
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1322550, completed 1322550, timed out 0, errs 0
MCU temperature: min 34.8, current 35.1, max 35.3
Supply voltage: min 24.0, current 24.1, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/1856/1420, gc cycles 61
Events: 1 queued, 1 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 0, ifcnt 207, reads 965, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 240, read errors 0, write errors 0, ifcnt 207, reads 965, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 160, reads 965, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 157, reads 966, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 158, reads 965, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-11-18 07:22:32
Cache data hit count 2212461605
Slowest loop: 10.21ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 3.6ms, write time 4.2ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 165, received 379, lost 0, errs 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 3, free buffers 26 (min 26), ts 92/92/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 20.33ms; fastest: 0.06ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 43120
WiFi IP address 192.168.10.x
Signal strength -53dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

---

But just out of curiosity, as I've understood it it's some DMA transfer between the CPU and WiFi-module that's causing the crashes.

Does this mean that if I've got a print I need/want to do I could just:

• Upload print job.
• Turn off the WiFi module with M552 S-1.
• Start the print from PanelDue, and let it complete.
• Once the job is done, turn on networking again from PanelDue with M552 S1.

Or will the DMA transfers still happen or try to happen, potentially causing a crash even if the WiFi-module is disabled?

[dc42]

@Exerqtor yes I think the crash will only occur when the WiFi module is enabled, because it's the code that talks to the WiFi module that is crashing.

[Exerqtor]

@dc42 Ok, good to know if I want to run a larger more time consuming print.

[dc42]

@Exerqtor please install the firmware at https://www.dropbox.com/scl/fo/nta4k70rxl6n0bsv8q4q0/h?rlkey=1oq6j3jrvoxa6uaofyn9re3nn&dl=0 and post the M122 after it next resets.

[Exerqtor]

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor please install the firmware at https://www.dropbox.com/scl/fo/nta4k70rxl6n0bsv8q4q0/h?rlkey=1oq6j3jrvoxa6uaofyn9re3nn&dl=0 and post the M122 after it next resets.

Will do!

[ClockWatchdog]

@Exerqtor I had the same errors with 3.4.5 - 3.4.6 and 3.5 Rc 1 with a duet 2 wifi.
I never had this kind of errors, it started to happen when i used a smartphone as a router. I suspected from the beginning, that the error was caused due to the continuous connect/reconnect. I "solved" (it is a workaround not a fix) using a PC as an Hotspot instead the phone and the crash never happened again. Later i inspected a bit, and with almost every phone or pad i used as an hotspot will cause connection-disconnection-reconnection continously and crashes.

[dc42]

@ClockWatchdog thanks, that's useful information. We know that the crash happens when something goes wrong during a transaction between the Duet and the WiFi module, but we haven't pinned it down yet.

[Exerqtor]

@dc42 new crash!

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1+ (2023-11-20 10:16:40) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 123844 of which 0 recycled
Never used RAM 11872, free system stack 186 words
Tasks: NETWORK(1,ready,17.4%,234) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,774) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,0.7%,108) MAIN(1,running,80.8%,670) IDLE(0,ready,0.3%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 01:06:22 ago, cause: software
Last software reset at 2023-11-21 16:39, reason: AssertionFailed, Gcodes spinning, available RAM 10964, slot 0
Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00446000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000905 000af288 0002ddb7 2002bed4 2002be01 000001a8 2002c100 20030c10 2002c118 2001e888 a5a5a5a5 a5a5a5a5 00000000 00000000 00000000 00000000 20030c1c 00000800 00000002 2002c100 00000002 2002bf9d 20018668 2001e888 0003002f 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 3983208, completed 3983208, timed out 0, errs 0
MCU temperature: min 34.8, current 35.3, max 38.2
Supply voltage: min 22.9, current 24.1, max 26.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/1400/964, gc cycles 185
Events: 1 queued, 1 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 1, ifcnt 30, reads 12987, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 246, read errors 0, write errors 1, ifcnt 30, reads 12987, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 218, reads 12986, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 216, reads 12986, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 216, reads 12987, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-11-21 17:45:13
Cache data hit count 4294967295
Slowest loop: 15.72ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 6.1ms, write time 4.5ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 35855, received 81685, lost 0, errs 0, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 264, free buffers 26 (min 25), ts 19914/19913/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 6.22ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 41332
WiFi IP address 192.168.10.x
Signal strength -55dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0 

[ClockWatchdog]

@dc42 I wondered, is it possible that the error is not exactly an error? I mean, when this disconnection-reconnection routine starts, is the number of interactions it performs too high and too concentrated? What if we reduced the speed of reconnection attempts and the number of attempts?

[NeoDue]

Hm. Reading through all this what your effort yielded, I wonder if whatever @dc42 might find here could also affect the (much smaller) issues I have on my Duet 3 6HC with wifi module

• dwell of about 5...10s required during boot prior to activating the module, otherwise the the module takes up to 10...15min to connect to the Wifi network
• random disconnects and
• very infrequent spontaneous reboots which I did not post here since I could not find anything useful about them yet.

I blamed the latter two on the web server and the firmware being beta in general - but who knows...?

[dc42]

@Exerqtor please try the firmware at https://www.dropbox.com/scl/fo/9hum0skl7rew21go56q7z/h?rlkey=mh1wx7mi72oz0dmk86c7uf4un&dl=0.

@NeoDue if you get any more spontaneous reboots of the Duet, please run M122 and post the output here.

[Exerqtor]

@dc42 New binaries installed, saw you did some changes to the WiFi interface in this fork.
Guess that means you're on to something? wishfull thinking 🙈

---

Just a little feedback after running the fw above for a while i've tried running a couple prints with a couple DWC instances open, and so far it haven't crashed at least. I'll try to have the printer printing for as many hours i can today with a bunch of DWC instances open to see if anything happens.

[Exerqtor]

@dc42 There she came, new crash:

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1+ (2023-11-24 09:09:58) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 10 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 124692 of which 12 recycled
Never used RAM 10388, free system stack 138 words
Tasks: NETWORK(2,nWait,17.0%,221) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,244) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,341) TMC(4,delaying,0.7%,80) MAIN(1,running,80.8%,670) IDLE(0,ready,0.6%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:08:14 ago, cause: software
Last software reset at 2023-11-25 16:29, reason: AssertionFailed, Gcodes spinning, available RAM 5228, slot 0
Software reset code 0x4923 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00487000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000907 000af288 0002dde9 2002bed4 2002be01 000001ae 2002c100 200303e0 2002c118 2001e888 0d362e30 a5a5a5a5 0a0d312e 00000000 00000000 00000000 200303ec 00000800 20035928 2002c100 20018668 2002bf9d 20018668 2001e888 0003007b 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 494787, completed 494785, timed out 0, errs 0
MCU temperature: min 45.3, current 45.3, max 45.8
Supply voltage: min 24.0, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/34, heap memory allocated/used/recyclable 2048/520/64, gc cycles 28
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 34, reads 863, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 0, ifcnt 34, reads 863, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 0, ifcnt 26, reads 862, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 0, ifcnt 27, reads 862, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 0, ifcnt 27, reads 863, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-11-25 16:37:43
Cache data hit count 846895387
Slowest loop: 10.63ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 3.6ms, write time 4.0ms, max retries 0
=== Move ===
DMs created 83, segments created 26, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 43, completed 43, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000807
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 147, received 333, lost 0, errs 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 6, free buffers 26 (min 26), ts 82/82/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 20.10ms; fastest: 0.07ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42968
WiFi IP address 192.168.10.x
Signal strength -49dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

---

EDIT 28.11.2023:
Was it any helpfull info in the M122 report @dc42? The printer has been idle since that crash, without any new ones.

[Exerqtor]

@dc42 New crash:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1+ (2023-11-24 09:09:58) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 123356 of which 0 recycled
Never used RAM 12360, free system stack 188 words
Tasks: NETWORK(1,ready,16.7%,219) HEAT(3,nWait,0.0%,327) Move(4,nWait,0.0%,348) CanReceiv(6,nWait,0.0%,774) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,0.7%,108) MAIN(1,running,81.5%,670) IDLE(0,ready,0.3%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:29:56 ago, cause: software
Last software reset at 2023-11-29 18:54, reason: HardFault imprec, Gcodes spinning, available RAM 10104, slot 1
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00000400 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fe8 Task NETW Freestk 506 ok
Stack: 2002c118 20030c48 200014e8 00000000 20032e7d 0002fef5 0002ffb4 610ff200 00034731 2002bed0 2002bed0 00000004 0002f635 200014e8 e000e000 00000020 0009fc25 00000000 20034fa8 00000000 20034fa8 00000000 df0bd10a e000e000 2001e888 a5a5a5a5 a5a5a5a5
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1796796, completed 1796794, timed out 0, errs 0
MCU temperature: min 32.9, current 33.3, max 36.5
Supply voltage: min 22.0, current 24.1, max 27.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/456/20, gc cycles 84
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 49, reads 29006, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 49, reads 29006, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 43, reads 29006, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 44, reads 29005, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 44, reads 29006, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-11-29 19:24:40
Cache data hit count 3045378646
Slowest loop: 12.84ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 6.7ms, write time 4.5ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 16181, received 36876, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 357, free buffers 26 (min 25), ts 8984/8983/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 6.04ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42796
WiFi IP address 192.168.10.x
Signal strength -47dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0 

[MaxGyver]

@Exerqtor @dc42

I can contribute some crash reports as well. My printer just rebooted mid print, this has never happend in one of the previous 3.5 betas.

Duet 3 6HC with SBC on RRF 3.5-rc.1

2.12.2023, 17:11:57: Verbindung zu duet3.local hergestellt
2.12.2023, 17:12:14: M201: Accelerations (mm/sec^2): X: 20000.0, Y: 20000.0, Z: 250.0, E: 20000.0
2.12.2023, 17:20:49: : Warning: SPI connection has been reset
2.12.2023, 17:20:51: : Error: Expansion board 123 reconnected
2.12.2023, 17:20:51: : Error: Expansion board 124 reconnected
2.12.2023, 17:20:52: : Error: Expansion board 126 reconnected
2.12.2023, 17:20:52: : Error: Expansion board 125 reconnected
2.12.2023, 17:20:55: : Warning: M307: Heater 0 predicted maximum temperature at full power is 321°C
2.12.2023, 17:20:55: : Warning: M307: Heater 1 predicted maximum temperature at full power is 534°C
2.12.2023, 17:20:55: : Warning: M307: Heater 1 predicted maximum temperature at full power is 534°C
2.12.2023, 17:20:56: : Warning: M307: Heater 1 predicted maximum temperature at full power is 534°C
2.12.2023, 17:21:07: M122: === Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.5.0-rc.1 (2023-08-31 16:19:24) running on Duet 3 MB6HC v1.0 or earlier (SBC mode)
Board ID: 08DJM-956L2-G43S8-6J1F2-3SN6N-1U0LG
Used output buffers: 1 of 40 (18 max)
=== RTOS ===
Static ram: 154852
Dynamic ram: 87280 of which 0 recycled
Never used RAM 3604, free system stack 204 words
Tasks: SBC(2,ready,1.7%,441) HEAT(3,nWait,0.0%,363) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,796) CanSender(5,nWait,0.0%,335) CanClock(7,delaying,0.0%,344) TMC(4,nWait,7.9%,61) MAIN(2,running,87.3%,137) IDLE(0,ready,3.1%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:00:17 ago, cause: software
Last software reset at 2023-12-02 16:20, reason: OutOfMemory, Gcodes spinning, available RAM 4, slot 0
Software reset code 0x41c3 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x20421688 Task Move Freestk 319 ok
Stack: 2041808c 00419f11 00000000 00000000 2045d0e0 0044a07d 204216f8 00442d47 2045d0e0 00000001 2045d0fc 00445b09 00000000 2045a200 204322a4 204296b6 204251bc 204296b8 00000000 00000002 00000000 20425cbc 00000001 00000001 20432418 00000000 00000001
Error status: 0x00
MCU temperature: min 34.5, current 34.8, max 35.2
Supply voltage: min 3.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 0.9, current 12.1, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/8, heap memory allocated/used/recyclable 2048/120/0, gc cycles 0
Events: 4 queued, 4 completed
Driver 0: standstill, SG min 0, mspos 8, reads 27152, writes 16 timeouts 0
Driver 1: standstill, SG min 0, mspos 8, reads 27153, writes 16 timeouts 0
Driver 2: standstill, SG min 0, mspos 8, reads 27153, writes 16 timeouts 0
Driver 3: standstill, SG min 0, mspos 8, reads 27153, writes 16 timeouts 0
Driver 4: standstill, SG min 0, mspos 8, reads 27151, writes 18 timeouts 0
Driver 5: standstill, SG min 0, mspos 8, reads 27158, writes 11 timeouts 0
Date/time: 2023-12-02 16:21:05
Slowest loop: 1.71ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 250, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x80000003
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 166, received 566, lost 0, boc 0
Longest wait 56ms for reply type 6041, peak Tx sync delay 38032, free buffers 50 (min 49), ts 90/86/0
Tx timeouts 0,0,3,0,0,0 last cancelled message type 30 dest 127
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 24333/3400
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x25a98
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server version 3.5.0-rc.1 (2023-09-01 07:16:31)
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 2
Full transfers per second: 49.12, max time between full transfers: 303.3ms, max pin wait times: 280.8ms/15.6ms
Codes per second: 65.98
Maximum length of RX/TX data transfers: 4456/1680