Random Hard Fault resets on Duet 3 Mini WiFi

[omtek]

Started a long print last night and woke up to two messages in the console:

*** Memory difference at line 2228 offset 4: original 392e303d copy 00000168, original changed, copy ok, fix=yes

*** Memory difference at line 2228 offset 4: original 392e303d copy 00000168, original changed, copy ok, fix=yes

(the same message occurs twice; potential crash caught?) Printer has been stable otherwise.

[Exerqtor]

Mine had a reset today while I was @work, but the only output in YAT is:

ok<LF>
HTTP is enabled on port 80<LF>
FTP is enabled on port 21<LF>
TELNET is disabled<LF>
CAN response timeout: board 121, req type 6054, RID 8<LF>
Warning: Heater 1 predicted maximum temperature at full power is 543°C<LF>
Log level is :  debug<LF>
Done!<LF>
RepRapFirmware for Duet 3 Mini 5+ is up and running.<LF>
WiFi module started<LF>
WiFi module is connected to access point IoT, IP address 192.168.30.x<LF>
HTTP is enabled on port 80<LF>
FTP is enabled on port 21<LF>
TELNET is disabled<LF>
CAN response timeout: board 121, req type 6054, RID 8<LF>
Warning: Heater 1 predicted maximum temperature at full power is 543°C<LF>
Log level is :  debug<LF>
Done!<LF>
RepRapFirmware for Duet 3 Mini 5+ is up and running.<LF>
WiFi module started<LF>
WiFi module is connected to access point IoT, IP address 192.168.30.x<LF>

Printed in one line on YAT, I just split it up for readabilty.

This is the M122 report:

2024-10-25 15:56:08 [debug] === Diagnostics ===
2024-10-25 15:56:08 [debug] RepRapFirmware for Duet 3 Mini 5+ version 3.5.3+1dbg (2024-10-20 14:13:46) running on Duet 3 Mini5plus WiFi (standalone mode)
2024-10-25 15:56:08 [debug] Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
2024-10-25 15:56:08 [debug] Used output buffers: 1 of 40 (39 max)
2024-10-25 15:56:08 [debug] === RTOS ===
2024-10-25 15:56:08 [debug] Static ram: 103368
2024-10-25 15:56:08 [debug] Dynamic ram: 123956 of which 0 recycled
2024-10-25 15:56:08 [debug] Never used RAM 11572, free system stack 142 words
2024-10-25 15:56:08 [debug] Tasks:
2024-10-25 15:56:08 [debug]  NETWORK(1,ready,18.9%,209)
2024-10-25 15:56:08 [debug]  HEAT(3,nWait 6,0.0%,325)
2024-10-25 15:56:08 [debug]  Move(4,nWait 6,0.0%,341)
2024-10-25 15:56:08 [debug]  CanReceiv(6,nWait 1,0.1%,773)
2024-10-25 15:56:08 [debug]  CanSender(5,nWait 7,0.0%,336)
2024-10-25 15:56:08 [debug]  CanClock(7,delaying,0.0%,348)
2024-10-25 15:56:08 [debug]  TMC(4,nWait 6,0.8%,101)
2024-10-25 15:56:08 [debug]  MAIN(1,running,78.5%,665)
2024-10-25 15:56:08 [debug]  IDLE(0,ready,0.8%,29)
2024-10-25 15:56:08 [debug]  AIN(4,delaying,0.8%,259)
2024-10-25 15:56:08 [debug] , total 100.0%
Owned mutexes:
2024-10-25 15:56:08 [debug]  WiFi(NETWORK)
2024-10-25 15:56:08 [debug]  USB(MAIN)
2024-10-25 15:56:08 [debug] === Platform ===
2024-10-25 15:56:08 [debug] Last reset 05:58:26 ago, cause: software
2024-10-25 15:56:08 [debug] Last software reset at 2024-10-25 09:57, reason: HardFault ibus, Gcodes spinning, available RAM 11596, slot 0
2024-10-25 15:56:08 [debug] Software reset code 0x0063 HFSR 0x40000000 CFSR 0x00000100 ICSR 0x00487803 BFAR 0xe000ed38 SP 0x20012038 Task NETW Freestk 494 ok
2024-10-25 15:56:08 [debug] Stack: 2002c620 200324a0 200014e4 00000000 20033689 000304b5 0d0a0d30 61010000 0d0a0d31 00000000 00000000 00000000 200120b8 00000014 000a0ea3 00000102 eeb20050 380aa8c0 080001ad 00000003 00034db1 2002c3b8 2002c3b8 00000001 0002fbf1 20011800 200114f8
2024-10-25 15:56:08 [debug] Error status: 0x00
2024-10-25 15:56:08 [debug] Aux0 errors 0,0,0
2024-10-25 15:56:08 [debug] MCU revision 3, ADC conversions started 16129917, completed 16129915, timed out 0, errs 0
2024-10-25 15:56:08 [debug] MCU temperature: min 35.5, current 35.9, max 38.8
2024-10-25 15:56:08 [debug] Supply voltage: min 3.8, current 24.2, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
2024-10-25 15:56:08 [debug] Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/580/152, gc cycles 4679
2024-10-25 15:56:08 [debug] Events: 1 queued, 1 completed
2024-10-25 15:56:08 [debug] Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 17785, writes 13, timeouts 0, DMA errors 0, CC errors 0
2024-10-25 15:56:08 [debug] Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 17784, writes 13, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x6f
2024-10-25 15:56:08 [debug] Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 50, reads 17783, writes 13, timeouts 0, DMA errors 0, CC errors 0
2024-10-25 15:56:08 [debug] Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 51, reads 17783, writes 13, timeouts 0, DMA errors 0, CC errors 0
2024-10-25 15:56:08 [debug] Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 17785, writes 13, timeouts 0, DMA errors 0, CC errors 0
2024-10-25 15:56:08 [debug] Driver 5: not present
2024-10-25 15:56:08 [debug] Driver 6: not present
2024-10-25 15:56:08 [debug] Date/time: 
2024-10-25 15:56:08 [debug] 2024-10-25 15:56:08
2024-10-25 15:56:08 [debug] Cache data hit count 4294967295
2024-10-25 15:56:08 [debug] Slowest loop: 13.34ms; fastest: 0.16ms
2024-10-25 15:56:08 [debug] === Storage ===
Free file entries: 18
2024-10-25 15:56:08 [debug] SD card 0 detected, interface speed: 22.5MBytes/sec
2024-10-25 15:56:08 [debug] SD card longest read time 8.0ms, write time 4.4ms, max retries 0
2024-10-25 15:56:08 [debug] === Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
2024-10-25 15:56:08 [debug] no step interrupt scheduled
2024-10-25 15:56:08 [debug] Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
2024-10-25 15:56:08 [debug] === DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
2024-10-25 15:56:08 [debug] === DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
2024-10-25 15:56:08 [debug] === Heat ===
2024-10-25 15:56:08 [debug] Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
2024-10-25 15:56:08 [debug] Heater 1 is on, I-accum = 0.0
2024-10-25 15:56:08 [debug] === GCodes ===
2024-10-25 15:56:08 [debug] Movement locks held by null, null
2024-10-25 15:56:08 [debug] HTTP is idle in state(s) 0
2024-10-25 15:56:08 [debug] Telnet is idle in state(s) 0
2024-10-25 15:56:08 [debug] File is idle in state(s) 0
2024-10-25 15:56:08 [debug] USB is ready with "m122" in state(s) 0
2024-10-25 15:56:08 [debug] Aux is idle in state(s) 0
2024-10-25 15:56:08 [debug] Trigger is idle in state(s) 0
2024-10-25 15:56:08 [debug] Queue is idle in state(s) 0
2024-10-25 15:56:08 [debug] LCD is idle in state(s) 0
2024-10-25 15:56:08 [debug] SBC is idle in state(s) 0
2024-10-25 15:56:08 [debug] Daemon is doing "G4 P250" in state(s) 0 0, running macro
2024-10-25 15:56:08 [debug] Aux2 is idle in state(s) 0
2024-10-25 15:56:08 [debug] Autopause is idle in state(s) 0
2024-10-25 15:56:08 [debug] File2 is idle in state(s) 0
2024-10-25 15:56:08 [debug] Queue2 is idle in state(s) 0
2024-10-25 15:56:08 [debug] Q0 segments left 0, axes/extruders owned 0x0000803
2024-10-25 15:56:08 [debug] Code queue 0 is empty
2024-10-25 15:56:08 [debug] Q1 segments left 0, axes/extruders owned 0x0000000
2024-10-25 15:56:08 [debug] Code queue 1 is empty
2024-10-25 15:56:08 [debug] === Filament sensors ===
check 0 clear 0
2024-10-25 15:56:08 [debug] Extruder 0 sensor: no filament
2024-10-25 15:56:08 [debug] === CAN ===
2024-10-25 15:56:08 [debug] Messages queued 193567, received 440879, lost 0, errs 383, boc 0
2024-10-25 15:56:08 [debug] Longest wait 2ms for reply type 6053, peak Tx sync delay 10353, free buffers 26 (min 25), ts 107533/107532/0
2024-10-25 15:56:08 [debug] Tx timeouts 0,0,0,0,0,0
2024-10-25 15:56:08 [debug] === Network ===
2024-10-25 15:56:08 [debug] Slowest loop: 13.86ms; fastest: 0.00ms
2024-10-25 15:56:08 [debug] Responder states:
2024-10-25 15:56:08 [debug]  MQTT(0)
2024-10-25 15:56:08 [debug]  HTTP(2)
2024-10-25 15:56:08 [debug]  HTTP(0)
2024-10-25 15:56:08 [debug]  HTTP(0)
2024-10-25 15:56:08 [debug]  HTTP(0)
2024-10-25 15:56:08 [debug]  FTP(0)
2024-10-25 15:56:08 [debug]  Telnet(0)
2024-10-25 15:56:08 [debug] HTTP sessions: 6 of 8
2024-10-25 15:56:08 [debug] === WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
2024-10-25 15:56:08 [debug] Firmware version 2.1.0
2024-10-25 15:56:08 [debug] MAC address c4:5b:be:ce:91:93
2024-10-25 15:56:08 [debug] Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 39616
2024-10-25 15:56:08 [debug] WiFi IP address 192.168.30.x
2024-10-25 15:56:08 [debug] Signal strength -51dBm, channel 6, mode 802.11n, reconnections 0
2024-10-25 15:56:08 [debug] Clock register 00002001
2024-10-25 15:56:08 [debug] Socket states:
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug] ok

[dc42]

@omtek yes, potential crash caught twice.

[omtek]

Three more messages overnight and this morning. They occurred after the printer was cooling down. Looks like another two potential crashes caught and averted. Still no resets to speak of.

*** Memory difference at line 2228 offset 4: original 392e303d copy 00000168, original changed, copy ok, fix=yes
*** Memory difference at line 2228 offset 52: original 20036658 copy 0d0a0d73, original ok, copy changed, fix=no
*** Memory difference at line 2228 offset 4: original 392e303d copy 00000168, original changed, copy ok, fix=yes

[Exerqtor]

Still haven't had one single event on 3.5.3-debug other than that one without any output. So I'm moving back over to 3.6.0-debug again so that I can get some prints done.
With nothing happening in well over 72hours I doubt anything will tbh.

[jovo0911]

After switching from the Maestro to the Mini 5 WiFi, my printer occasionally stopped or crashed without any apparent reason. I found this thread and installed the 3.6.0-debug firmware from dc42. Since then, I’ve completed over ten prints, including some taller than 150mm and others over 4 hours long, with no crashes. My printer is a CoreXY setup with dual Z-axis and a single extruder.

[dc42]

@jovo0911 thanks. I've established that this issue does not cause the same type of reset in 3.6.0 beta1 that it does in 3.5.x and 3.4.x. What I don't know is whether it causes any other issues in 3.6.0 beta1. I've also developed a workaround for the issue in 3.5.3.

[jovo0911]

Should I go back to 3.4.6 and wait? (The Maestro ran fine with that version.) I started with 3.5.0 on the Mini 5 and initially thought the board might have a hardware issue.
But now the printer is running smoothly with the 3.6.0-debug firmware.

[dc42]

@jovo0911 the issue discussed in this thread is unique to the Duet 3 Mini WiFi.

[wschadow]

@dc42 Here is some feedback from my side. For about a week I tried 3.6.0-beta1-2 on a Duet3Mini5 where everything was connected directly to the board. Good news: WiFi and printer ran without error. Observations:

1. YAP connects, but no data is displayed in YAP. M115 does not return anything.

2. The old home sequence doesn't work without change

; =========================================================================================================
; home y axis
; =========================================================================================================
;
M400                                                                   ; wait for current moves to finish
G1 H2 Y0.5 F1000                                                       ; energise motors to ensure they are not stalled
M400                                                                   ; wait for current moves to finish
G1 H1 Y5 F1000                                                         ; move slowly away
G1 H1 Y-260 F1800                                                      ; move quickly to x axis endstop and stop there (first pass)
G1 H2 Y5 F1000                                                         ; go back a few mm
G1 H1 Y-10 F1800                                                       ; move slowly to x axis endstop once more (second pass)

G28
Error: G1: Homing move abandoned because driver 3 is moving too slowly for stall detection

It had to be changed to

; =========================================================================================================
; home y axis
; =========================================================================================================
;
M400                                                                   ; wait for current moves to finish
G1 H2 Y0.5 F2000                                                       ; energise motors to ensure they are not stalled
M400                                                                   ; wait for current moves to finish
G1 H1 Y5 F2000                                                         ; move slowly away
G1 H1 Y-260 F1800                                                      ; move quickly to x axis endstop and stop there (first pass)
G1 H2 Y5 F2000                                                         ; go back a few mm
G1 H1 Y-10 F1800                                                       ; move slowly to x axis endstop once more (second pass)
;

[wschadow]

@dc42 Today I tried 3.6.0-beta1-2 on a Dual extruder where the extruders have a can board. So far, this works as well.