Random Hard Fault resets on Duet 3 Mini WiFi

omtek

@dc42 Logged another message. Printer was idle at the time, and this was the first memory difference message since my last post. There haven't been any resets since my last post, either.

*** Memory difference at offset 20: was 2001ea80 now 00000000

omtek

Had another reset this morning. I didn't record any additional memory difference messages. The printer was sitting idle, cooling down after completing a print about 35-40 minutes prior.

M122

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.3+1dbg (2024-10-14 09:45:56) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: NTK2Z-J196U-D65J0-40KMN-1V03Z-ZJMGF
Used output buffers: 3 of 40 (28 max)
=== RTOS ===
Static ram: 103368
Dynamic ram: 124564 of which 12 recycled
Never used RAM 10952, free system stack 156 words
Tasks: NETWORK(2,nWait 7,15.5%,240) LASER(5,nWait 7,0.0%,269) HEAT(3,nWait 6,0.0%,325) Move(4,nWait 6,0.0%,341) CanReceiv(6,nWait 1,0.1%,773) CanSender(5,nWait 7,0.0%,336) CanClock(7,delaying,0.0%,348) TMC(4,delaying,1.5%,101) MAIN(1,running,82.1%,665) IDLE(0,ready,0.0%,29) AIN(4,delaying,0.8%,259), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 03:15:48 ago, cause: software
Last software reset at 2024-10-20 05:47, reason: HardFault bfarValid precise, Expansion spinning, available RAM 7204, slot 0
Software reset code 0x4072 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00446803 BFAR 0x00000004 SP 0x20012008 Task NETW Freestk 482 ok
Stack: 00000167 00000002 200014e4 00000166 20033af3 000a102f 0003041a 810f0000 00030411 00000000 00000000 00000000 20032964 00000800 20036670 2002c5d8 2001882c 2002c42e 2001882c 2001ea80 000305ab 00000000 00000000 00000000 200120b8 00000014 b5ddddb2
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 8811290, completed 8811289, timed out 0, errs 0
MCU temperature: min 33.0, current 33.3, max 35.4
Supply voltage: min 24.3, current 24.4, max 24.5, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/10, heap memory allocated/used/recyclable 2048/304/132, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 115, reads 19406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 113, reads 19406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 65, reads 19409, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 120, reads 19406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 119, reads 19406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 2, read errors 0, write errors 1, ifcnt 113, reads 19406, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 65, reads 19409, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-10-20 09:02:56
Cache data hit count 4294967295
Slowest loop: 9.39ms; fastest: 0.16ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.8ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 105742, received 334856, lost 0, errs 0, boc 0
Longest wait 2ms for reply type 6060, peak Tx sync delay 284, free buffers 26 (min 25), ts 58741/58740/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 13.05ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1.0
MAC address d8:bf:c0:14:e5:fc
Module reset reason: Power up, Vcc 3.36, flash size 2097152, free heap 42764
WiFi IP address 192.168.2.8
Signal strength -51dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

dc42

@omtek thanks again. Based on your results and some messages from my own system, I have built up a picture of what is happening.

@omtek @Exerqtor @wschadow @Maurits @Inlinebrother please install the new 3.5.3 debug firmware that I have just put at https://www.dropbox.com/scl/fo/vp24fllwuua9yejr8pej1/ANRHyUhbMU5DRTJBNeF1sKg?rlkey=97mm1rpm4h0m22bjalgxmrrwq&dl=0. Like the previous version, this will print a debug message to USB when it detects that something is amiss. This build also attempts to recover from the situation in order to avoid a reset.

As before, please leave this running on your machine with a PC running a terminal emulator connected to the Duet USB port, and at least one instance of DWC connected to it via WiFi. Please report any "Memory difference" messages appearing on that terminal, also the M122 report if you are unlucky enough to have a reset.

Based on your reports and any further messages that my own system produces, I will evaluate whether this code with the debug removed can form the basis of a solution to this annoying issue.

omtek

@dc42 Logged a memory difference message with the new build. Printer was idle. No reset as of yet.

*** Memory difference at line 2228 offset 12: original 0d0a0d39 copy 20032958, original changed, copy ok, fix=yes

Logged another memory difference message. Still no reset.

*** Memory difference at line 2228 offset 60: original 2001882c copy 0d0a0d39, original ok, copy changed, fix=no

Exerqtor

@dc42 Haven't had time to look at this before now (like I mentioned in the email), but do you have a debug version based on 3.6 as well?

dc42

@Exerqtor I'll prepare a 3.6 version with the debug in it today.

dc42

@Exerqtor I've put a 3.6.0-beta.1+debug build for the Duet 3 Mini at https://www.dropbox.com/scl/fo/01s6pikmcusa2e3vu0rkh/APdYxDkXDSIWqeQl0R__Gag?rlkey=iwd3zb2dk3ggjw7ihcy2whg5h&dl=0. The instructions are slightly different from the 3.5.3 version:

Install it
Preferably, connect a PC running a terminal emulator to the USB port, check it is communicating with the Duet, and execute M111 P8 S1 to enable debug messages
Leave it running with at least one DWC session connected
Report any "Mem diff" messages that are displayed on the terminal emulator
If you get any unexpected resets, post a M122 report

If you don't wish to connect a terminal emulator, then don't run M111 P8 S1 and it won't attempt to output the debug messages to USB. It will still log up to 4 memory difference reports internally, but only if the board doesn't reset. Running M122 will report these messages just before the "Storage" section of the report, and clear them.

dc42

@omtek thanks, that's useful information. The first memory difference would likely have resulted in a reset in previous firmware versions, but the memory check detected and fixed it. The second memory difference was harmless.

Exerqtor

@omtek said in Random Hard Fault resets on Duet 3 Mini WiFi:

@dc42 Logged a memory difference message with the new build. Printer was idle. No reset as of yet.

Out of curiosity, how long had you had the logging going before it threw these messages?

omtek

@Exerqtor With both debug builds I logged a memory difference message within a couple of hours. The first debug build was under an hour, the second took maybe 1-2 hours to show up.

dc42

For comparison, my system has logged anything from 0 to 4 memory differences in 24 hours. I have two instances of DWC connected (from Firefox and Chrome running on the same PC) and I reduced the DWC polling interval to 100ms.

Exerqtor

@omtek said in Random Hard Fault resets on Duet 3 Mini WiFi:

@Exerqtor With both debug builds I logged a memory difference message within a couple of hours. The first debug build was under an hour, the second took maybe 1-2 hours to show up.

@dc42 said in Random Hard Fault resets on Duet 3 Mini WiFi:

For comparison, my system has logged anything from 0 to 4 memory differences in 24 hours. I have two instances of DWC connected (from Firefox and Chrome running on the same PC) and I reduced the DWC polling interval to 100ms.

Hmm ok, in that case my machine is either really good behaved or I've setup something wrong

Installed the 3.6-debug fw.
Hooked a computer with USB.
Installed the newest YAT from sourceforge.
Connected it to the printer (tested by sending G28 in the termnial and the printer homed).
After testing connection I sent M111 P8 S1 through YAT.
Opened three instances(not three tabs, if thats any difference) or DWC in chrome + one instance of OrcaSlicer on one pc, plus i check in on my phone every once i a while too.

It's now been sitting for roughly 22 hours without anything happening. No output in YAT, and no reboot/reset.

gloomyandy

@Exerqtor Did you...

Leave it running with at least one DWC session connected

I think DC42 said he had two DWC instances running (in different browsers).

Exerqtor

@gloomyandy Added that info to the previous post 🙈

dc42

@Exerqtor have you ever had one of these resets since you upgraded to 3.6 beta 1?

Exerqtor

@dc42 Not really, after I stopped using chrome (other than the occasionall check-in with chrome on my phone) I haven't really had any resets at all that I've noticed.

The laptop have restarted (forced Windows update 🤦‍♂️) within the last two hours. So i don't know if anything got output in that timespan (no resets though). I've got it back up logging now and plan to have it runnning until the weekend.

I see the debug log has reached 3.21gb by now though. So idk if I maybe should have disabled that.

Going on 22hours since the laptop rebooted, and it's still no output on my end.

omtek

Another memory difference message this afternoon. No resets to speak of, either.

*** Memory difference at line 2228 offset 12: original 0a0d392e copy 20032128, original changed, copy ok, fix=yes

and another

*** Memory difference at line 2228 offset 52: original 20036658 copy 0d0a0d6d, original ok, copy changed, fix=no

edit #3 - busy day...

*** Memory difference at line 2228 offset 60: original 2001882c copy 0a0d392e, original ok, copy changed, fix=no

omtek

Two more memory difference messages logged overnight. Printer was idle but has been printing nicely. Still no resets to speak of.

*** Memory difference at line 2228 offset 60: original 2001882c copy 0a0d392e, original ok, copy changed, fix=no

*** Memory difference at line 2228 offset 56: original 2002c5d8 copy 0a0d656e, original ok, copy changed, fix=no

Exerqtor

No output in my end what so ever. Has anyone else had outputs with the 3.6 build? 🤔

dc42

@Exerqtor I had no output or resets using 3.6 either. I reverted to 3.5.3 and got two memory difference reports within a few hours.