Random Hard Fault resets on Duet 3 Mini WiFi

dc42

@Exerqtor I had no output or resets using 3.6 either. I reverted to 3.5.3 and got two memory difference reports within a few hours.

Exerqtor

~~@dc42~~ ~~Ok, should i do that too? At least until tomorrow, or do you have enough datapoints on 3.5?~~

Reverted to 3.5.3 debug and have it running now. Straight off the bat I noticed a difference, when I sent G28Z in YAT to comfirm connection i got ok<LF> in return, but I didn't get anything in return when I input something in the 3.6 debug.

oliof

@Exerqtor do your machines run in Marlin compat mode?

dc42

@Exerqtor bear in mind that recent versions of YAT do not reconnect automatically when the Duet restarts. I always have to disconnect and reconnect YAT manually using the icons in the menu bar.

Chriss

@dc42
Let me point out one more thing here: The printer I see this problem does not have this problem since some weeks anymore. But all of the print which the printer performed are below 100mm in Z. I'm not sure whether that is time ore high related. I just wanted to let you know my observation.

Exerqtor

@oliof Uuuuhm, nope:

25.10.2024, 06:51:35	M555
Output mode: RepRapFirmware

I didn't know it was a thing even 😆

@dc42 said in Random Hard Fault resets on Duet 3 Mini WiFi:

@Exerqtor bear in mind that recent versions of YAT do not reconnect automatically when the Duet restarts. I always have to disconnect and reconnect YAT manually using the icons n the menu bar.

Yeah i've made sure to home the printer, and it's still homed. So it haven't been any resets 😅

omtek

Started a long print last night and woke up to two messages in the console:

*** Memory difference at line 2228 offset 4: original 392e303d copy 00000168, original changed, copy ok, fix=yes

(the same message occurs twice; potential crash caught?) Printer has been stable otherwise.

Exerqtor

Mine had a reset today while I was @work, but the only output in YAT is:

ok<LF>
HTTP is enabled on port 80<LF>
FTP is enabled on port 21<LF>
TELNET is disabled<LF>
CAN response timeout: board 121, req type 6054, RID 8<LF>
Warning: Heater 1 predicted maximum temperature at full power is 543°C<LF>
Log level is :  debug<LF>
Done!<LF>
RepRapFirmware for Duet 3 Mini 5+ is up and running.<LF>
WiFi module started<LF>
WiFi module is connected to access point IoT, IP address 192.168.30.x<LF>
HTTP is enabled on port 80<LF>
FTP is enabled on port 21<LF>
TELNET is disabled<LF>
CAN response timeout: board 121, req type 6054, RID 8<LF>
Warning: Heater 1 predicted maximum temperature at full power is 543°C<LF>
Log level is :  debug<LF>
Done!<LF>
RepRapFirmware for Duet 3 Mini 5+ is up and running.<LF>
WiFi module started<LF>
WiFi module is connected to access point IoT, IP address 192.168.30.x<LF>

Printed in one line on YAT, I just split it up for readabilty.

This is the M122 report:

2024-10-25 15:56:08 [debug] === Diagnostics ===
2024-10-25 15:56:08 [debug] RepRapFirmware for Duet 3 Mini 5+ version 3.5.3+1dbg (2024-10-20 14:13:46) running on Duet 3 Mini5plus WiFi (standalone mode)
2024-10-25 15:56:08 [debug] Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
2024-10-25 15:56:08 [debug] Used output buffers: 1 of 40 (39 max)
2024-10-25 15:56:08 [debug] === RTOS ===
2024-10-25 15:56:08 [debug] Static ram: 103368
2024-10-25 15:56:08 [debug] Dynamic ram: 123956 of which 0 recycled
2024-10-25 15:56:08 [debug] Never used RAM 11572, free system stack 142 words
2024-10-25 15:56:08 [debug] Tasks:
2024-10-25 15:56:08 [debug]  NETWORK(1,ready,18.9%,209)
2024-10-25 15:56:08 [debug]  HEAT(3,nWait 6,0.0%,325)
2024-10-25 15:56:08 [debug]  Move(4,nWait 6,0.0%,341)
2024-10-25 15:56:08 [debug]  CanReceiv(6,nWait 1,0.1%,773)
2024-10-25 15:56:08 [debug]  CanSender(5,nWait 7,0.0%,336)
2024-10-25 15:56:08 [debug]  CanClock(7,delaying,0.0%,348)
2024-10-25 15:56:08 [debug]  TMC(4,nWait 6,0.8%,101)
2024-10-25 15:56:08 [debug]  MAIN(1,running,78.5%,665)
2024-10-25 15:56:08 [debug]  IDLE(0,ready,0.8%,29)
2024-10-25 15:56:08 [debug]  AIN(4,delaying,0.8%,259)
2024-10-25 15:56:08 [debug] , total 100.0%
Owned mutexes:
2024-10-25 15:56:08 [debug]  WiFi(NETWORK)
2024-10-25 15:56:08 [debug]  USB(MAIN)
2024-10-25 15:56:08 [debug] === Platform ===
2024-10-25 15:56:08 [debug] Last reset 05:58:26 ago, cause: software
2024-10-25 15:56:08 [debug] Last software reset at 2024-10-25 09:57, reason: HardFault ibus, Gcodes spinning, available RAM 11596, slot 0
2024-10-25 15:56:08 [debug] Software reset code 0x0063 HFSR 0x40000000 CFSR 0x00000100 ICSR 0x00487803 BFAR 0xe000ed38 SP 0x20012038 Task NETW Freestk 494 ok
2024-10-25 15:56:08 [debug] Stack: 2002c620 200324a0 200014e4 00000000 20033689 000304b5 0d0a0d30 61010000 0d0a0d31 00000000 00000000 00000000 200120b8 00000014 000a0ea3 00000102 eeb20050 380aa8c0 080001ad 00000003 00034db1 2002c3b8 2002c3b8 00000001 0002fbf1 20011800 200114f8
2024-10-25 15:56:08 [debug] Error status: 0x00
2024-10-25 15:56:08 [debug] Aux0 errors 0,0,0
2024-10-25 15:56:08 [debug] MCU revision 3, ADC conversions started 16129917, completed 16129915, timed out 0, errs 0
2024-10-25 15:56:08 [debug] MCU temperature: min 35.5, current 35.9, max 38.8
2024-10-25 15:56:08 [debug] Supply voltage: min 3.8, current 24.2, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
2024-10-25 15:56:08 [debug] Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/580/152, gc cycles 4679
2024-10-25 15:56:08 [debug] Events: 1 queued, 1 completed
2024-10-25 15:56:08 [debug] Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 17785, writes 13, timeouts 0, DMA errors 0, CC errors 0
2024-10-25 15:56:08 [debug] Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 17784, writes 13, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x6f
2024-10-25 15:56:08 [debug] Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 50, reads 17783, writes 13, timeouts 0, DMA errors 0, CC errors 0
2024-10-25 15:56:08 [debug] Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 51, reads 17783, writes 13, timeouts 0, DMA errors 0, CC errors 0
2024-10-25 15:56:08 [debug] Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 17785, writes 13, timeouts 0, DMA errors 0, CC errors 0
2024-10-25 15:56:08 [debug] Driver 5: not present
2024-10-25 15:56:08 [debug] Driver 6: not present
2024-10-25 15:56:08 [debug] Date/time: 
2024-10-25 15:56:08 [debug] 2024-10-25 15:56:08
2024-10-25 15:56:08 [debug] Cache data hit count 4294967295
2024-10-25 15:56:08 [debug] Slowest loop: 13.34ms; fastest: 0.16ms
2024-10-25 15:56:08 [debug] === Storage ===
Free file entries: 18
2024-10-25 15:56:08 [debug] SD card 0 detected, interface speed: 22.5MBytes/sec
2024-10-25 15:56:08 [debug] SD card longest read time 8.0ms, write time 4.4ms, max retries 0
2024-10-25 15:56:08 [debug] === Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
2024-10-25 15:56:08 [debug] no step interrupt scheduled
2024-10-25 15:56:08 [debug] Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
2024-10-25 15:56:08 [debug] === DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
2024-10-25 15:56:08 [debug] === DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
2024-10-25 15:56:08 [debug] === Heat ===
2024-10-25 15:56:08 [debug] Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
2024-10-25 15:56:08 [debug] Heater 1 is on, I-accum = 0.0
2024-10-25 15:56:08 [debug] === GCodes ===
2024-10-25 15:56:08 [debug] Movement locks held by null, null
2024-10-25 15:56:08 [debug] HTTP is idle in state(s) 0
2024-10-25 15:56:08 [debug] Telnet is idle in state(s) 0
2024-10-25 15:56:08 [debug] File is idle in state(s) 0
2024-10-25 15:56:08 [debug] USB is ready with "m122" in state(s) 0
2024-10-25 15:56:08 [debug] Aux is idle in state(s) 0
2024-10-25 15:56:08 [debug] Trigger is idle in state(s) 0
2024-10-25 15:56:08 [debug] Queue is idle in state(s) 0
2024-10-25 15:56:08 [debug] LCD is idle in state(s) 0
2024-10-25 15:56:08 [debug] SBC is idle in state(s) 0
2024-10-25 15:56:08 [debug] Daemon is doing "G4 P250" in state(s) 0 0, running macro
2024-10-25 15:56:08 [debug] Aux2 is idle in state(s) 0
2024-10-25 15:56:08 [debug] Autopause is idle in state(s) 0
2024-10-25 15:56:08 [debug] File2 is idle in state(s) 0
2024-10-25 15:56:08 [debug] Queue2 is idle in state(s) 0
2024-10-25 15:56:08 [debug] Q0 segments left 0, axes/extruders owned 0x0000803
2024-10-25 15:56:08 [debug] Code queue 0 is empty
2024-10-25 15:56:08 [debug] Q1 segments left 0, axes/extruders owned 0x0000000
2024-10-25 15:56:08 [debug] Code queue 1 is empty
2024-10-25 15:56:08 [debug] === Filament sensors ===
check 0 clear 0
2024-10-25 15:56:08 [debug] Extruder 0 sensor: no filament
2024-10-25 15:56:08 [debug] === CAN ===
2024-10-25 15:56:08 [debug] Messages queued 193567, received 440879, lost 0, errs 383, boc 0
2024-10-25 15:56:08 [debug] Longest wait 2ms for reply type 6053, peak Tx sync delay 10353, free buffers 26 (min 25), ts 107533/107532/0
2024-10-25 15:56:08 [debug] Tx timeouts 0,0,0,0,0,0
2024-10-25 15:56:08 [debug] === Network ===
2024-10-25 15:56:08 [debug] Slowest loop: 13.86ms; fastest: 0.00ms
2024-10-25 15:56:08 [debug] Responder states:
2024-10-25 15:56:08 [debug]  MQTT(0)
2024-10-25 15:56:08 [debug]  HTTP(2)
2024-10-25 15:56:08 [debug]  HTTP(0)
2024-10-25 15:56:08 [debug]  HTTP(0)
2024-10-25 15:56:08 [debug]  HTTP(0)
2024-10-25 15:56:08 [debug]  FTP(0)
2024-10-25 15:56:08 [debug]  Telnet(0)
2024-10-25 15:56:08 [debug] HTTP sessions: 6 of 8
2024-10-25 15:56:08 [debug] === WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
2024-10-25 15:56:08 [debug] Firmware version 2.1.0
2024-10-25 15:56:08 [debug] MAC address c4:5b:be:ce:91:93
2024-10-25 15:56:08 [debug] Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 39616
2024-10-25 15:56:08 [debug] WiFi IP address 192.168.30.x
2024-10-25 15:56:08 [debug] Signal strength -51dBm, channel 6, mode 802.11n, reconnections 0
2024-10-25 15:56:08 [debug] Clock register 00002001
2024-10-25 15:56:08 [debug] Socket states:
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug]  0
2024-10-25 15:56:08 [debug] ok

dc42

@omtek yes, potential crash caught twice.

omtek

Three more messages overnight and this morning. They occurred after the printer was cooling down. Looks like another two potential crashes caught and averted. Still no resets to speak of.

*** Memory difference at line 2228 offset 4: original 392e303d copy 00000168, original changed, copy ok, fix=yes
*** Memory difference at line 2228 offset 52: original 20036658 copy 0d0a0d73, original ok, copy changed, fix=no
*** Memory difference at line 2228 offset 4: original 392e303d copy 00000168, original changed, copy ok, fix=yes

Exerqtor

Still haven't had one single event on 3.5.3-debug other than that one without any output. So I'm moving back over to 3.6.0-debug again so that I can get some prints done.
With nothing happening in well over 72hours I doubt anything will tbh.

jovo0911

After switching from the Maestro to the Mini 5 WiFi, my printer occasionally stopped or crashed without any apparent reason. I found this thread and installed the 3.6.0-debug firmware from dc42. Since then, I’ve completed over ten prints, including some taller than 150mm and others over 4 hours long, with no crashes. My printer is a CoreXY setup with dual Z-axis and a single extruder.

dc42

@jovo0911 thanks. I've established that this issue does not cause the same type of reset in 3.6.0 beta1 that it does in 3.5.x and 3.4.x. What I don't know is whether it causes any other issues in 3.6.0 beta1. I've also developed a workaround for the issue in 3.5.3.

jovo0911

Should I go back to 3.4.6 and wait? (The Maestro ran fine with that version.) I started with 3.5.0 on the Mini 5 and initially thought the board might have a hardware issue.
But now the printer is running smoothly with the 3.6.0-debug firmware.

dc42

@jovo0911 the issue discussed in this thread is unique to the Duet 3 Mini WiFi.

wschadow

@dc42 Here is some feedback from my side. For about a week I tried 3.6.0-beta1-2 on a Duet3Mini5 where everything was connected directly to the board. Good news: WiFi and printer ran without error. Observations:

YAP connects, but no data is displayed in YAP. M115 does not return anything.
The old home sequence doesn't work without change

; =========================================================================================================
; home y axis
; =========================================================================================================
;
M400                                                                   ; wait for current moves to finish
G1 H2 Y0.5 F1000                                                       ; energise motors to ensure they are not stalled
M400                                                                   ; wait for current moves to finish
G1 H1 Y5 F1000                                                         ; move slowly away
G1 H1 Y-260 F1800                                                      ; move quickly to x axis endstop and stop there (first pass)
G1 H2 Y5 F1000                                                         ; go back a few mm
G1 H1 Y-10 F1800                                                       ; move slowly to x axis endstop once more (second pass)

G28
Error: G1: Homing move abandoned because driver 3 is moving too slowly for stall detection

It had to be changed to

; =========================================================================================================
; home y axis
; =========================================================================================================
;
M400                                                                   ; wait for current moves to finish
G1 H2 Y0.5 F2000                                                       ; energise motors to ensure they are not stalled
M400                                                                   ; wait for current moves to finish
G1 H1 Y5 F2000                                                         ; move slowly away
G1 H1 Y-260 F1800                                                      ; move quickly to x axis endstop and stop there (first pass)
G1 H2 Y5 F2000                                                         ; go back a few mm
G1 H1 Y-10 F1800                                                       ; move slowly to x axis endstop once more (second pass)
;

wschadow

@dc42 Today I tried 3.6.0-beta1-2 on a Dual extruder where the extruders have a can board. So far, this works as well.

Inlinebrother

Hi, don't know if it is still needed but I am still running 3.5.3 and had a hardfault today. I see that there is an update with bug fixes. Is there a fix for this or is it still in development?

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.3 (2024-09-18 11:25:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: B7FWR-UK9MK-K65J0-40TKG-0ZW2Z-RKN9N
Used output buffers: 1 of 40 (19 max)
=== RTOS ===
Static ram: 103368
Dynamic ram: 122076 of which 12 recycled
Never used RAM 16292, free system stack 202 words
Tasks: NETWORK(2,nWait 7,14.2%,217) HEAT(3,nWait 1,0.0%,327) Move(4,nWait 6,0.0%,355) CanReceiv(6,nWait 1,0.0%,939) CanSender(5,nWait 7,0.0%,336) CanClock(7,delaying,0.0%,334) TMC(4,delaying,1.4%,110) MAIN(1,running,83.5%,665) IDLE(0,ready,0.0%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:16:08 ago, cause: software
Last software reset at 2024-12-26 22:08, reason: HardFault bfarValid precise, Platform spinning, available RAM 12488, slot 1
Software reset code 0x4060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00000803 BFAR 0x0000001c SP 0x20012008 Task NETW Freestk 482 ok
Stack: 2002c658 200306a0 200014e4 00000000 ffffffff 0003039d 000302b0 610f0000 00030389 00000000 00000000 00000000 200306ac 00000800 20035c08 2002c640 2001882c 2002c496 2001882c 2001ea80 00030523 00000000 00000000 00000000 200120b8 00000014 b5ddd790
Error status: 0x00
MCU revision 3, ADC conversions started 726187, completed 726185, timed out 0, errs 0
MCU temperature: min 35.4, current 35.6, max 38.9
Supply voltage: min 24.2, current 24.3, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 28, reads 22449, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 28, reads 22449, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 21, reads 22449, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 60, read errors 0, write errors 1, ifcnt 29, reads 22449, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 21, reads 22449, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 2, read errors 0, write errors 1, ifcnt 24, reads 22449, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 21, reads 22449, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-12-26 22:24:38
Cache data hit count 1821230026
Slowest loop: 5.84ms; fastest: 0.16ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.7ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 8712, received 0, lost 0, errs 4576139, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 26 (min 26), ts 4841/0/0
Tx timeouts 0,0,4840,0,0,3870 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 15.79ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 1.25
MAC address bc:ff:4d:e7:aa:41
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 26416
WiFi IP address 192.168.1.146
Signal strength -51dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0


M122 P106
Platform 2001d138-2001e737
SbcInterface 2001e740-2001ea70
Network 2001ea80-2001eb67
GCodes 2001eb70-2001ff6f
Move 20023ff8-20024d33
Heat 20025110-20025237
PrintMonitor 20025240-2002544f
FansManager 20025458-200254d7
PortControl 200254e0-2002550f
Display 20025518-20025533
ExpansionManager 20025540-200282f7

m122 p1007 a{0x2001ea80+8}
2001ea88: 2002c420

m122 p1007 a{0x2002c420+136}
2002c4a8: 20032f20

dc42

@Inlinebrother this was fixed in 3.5.4.