Duet 2 WiFi HardFault invState

Leonard03

Hello everyone!
Sadly, I have again problems with my board which started to restart itself after many hours during a print

My new reason for the resets is "HardFault invState".
SD card image/configuration is here: https://drive.google.com/drive/u/0/folders/1OcTPiUw0RlYuLT-Skopb6It37nG4u9Io

The resets occured one time after 4 hours into a print and another after 7,5 hours the reason being the same. Both times the WiFi network/DWC page has long gone (maybe after one hour after starting a print)

The response from M122 is:

10/11/2023, 1:57:40 PM: Connected to 192.168.0.107
10/11/2023, 4:19:25 PM: M122: === Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.5.0-rc.1+ (2023-09-27 13:34:46) running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-917NK-F2MS4-7JKDG-3S06M-9ZSWD
Used output buffers: 3 of 26 (26 max)
=== RTOS ===
Static ram: 23076
Dynamic ram: 77804 of which 12 recycled
Never used RAM 9396, free system stack 194 words
Tasks: NETWORK(2,nWait,14.0%,218) HEAT(3,nWait,0.0%,329) Move(4,nWait,0.0%,364) DUEX(5,nWait,0.0%,26) MAIN(1,running,85.9%,743) IDLE(0,ready,0.0%,29), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:21:59 ago, cause: software
Last software reset at 2023-10-11 15:57, reason: HardFault invState, Gcodes spinning, available RAM 6372, slot 0
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x0041f803 BFAR 0xe000ed38 SP 0x200022b8 Task MAIN Freestk 995 ok
Stack: 00000001 200012d4 20000150 06000000 00000000 0045ef43 200048dc 000f0000 200048dc 00000003 00000003 00000000 00000003 0000003e 2000239c 00000000 004592f9 00000000 00433ed9 2000239c 0000003e 2000239c 00000003 00000000 200049a5 2000241e 00000001
Error status: 0x04
Aux0 errors 0,0,0
MCU temperature: min 25.1, current 25.1, max 26.2
Supply voltage: min 0.5, current 0.6, max 0.6, under voltage events: 0, over voltage events: 0, power good: no
Heap OK, handles allocated/used 99/23, heap memory allocated/used/recyclable 2048/504/44, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok, SG min n/a
Driver 1: ok, SG min n/a
Driver 2: ok, SG min n/a
Driver 3: ok, SG min n/a
Driver 4: ok, SG min n/a
Driver 5: ok, SG min n/a
Driver 6: ok, SG min n/a
Driver 7: ok, SG min n/a
Driver 8: ok, SG min n/a
Driver 9: ok, SG min n/a
Driver 10: 
Driver 11: 
Date/time: 2023-10-11 16:19:26
Cache data hit count 4294967295
Slowest loop: 9.76ms; fastest: 0.15ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 4.9ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Q0 segments left 0
Code queue 0 is empty
=== DueX ===
Read count 0, 0.00 reads/min
=== Network ===
Slowest loop: 21.31ms; fastest: 0.07ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address b4:e6:2d:52:f5:47
Module reset reason: Power up, Vcc 3.40, flash size 2097152, free heap 42864
WiFi IP address 192.168.0.107
Signal strength -34dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

Searching the .map file, I found this codes from the stacktrace:

00000001	 .data.tzinfo   0x20000158       0x58 c:/program files (x86)/arm gnu toolchain arm-none-eabi/12.2 mpacbti-rel1/bin/../lib/gcc/arm-none-eabi/12.2.1/thumb/v7e-m+fp/hard\libc.a(libc_a-gettzinfo.o)
                0x200001b0                        . = ALIGN (0x4)
                0x200001b0                        _erelocate = .
                0x0047f228                        _firmware_end = (_etext + (_erelocate - _srelocate))
                0x0047f228                        _firmware_crc = _firmware_end
                0x00000001                        ASSERT (((_firmware_crc + 0x4) <= (ORIGIN (rom) + LENGTH (rom))), region ROM overflowed)

20000150	.data.uxCriticalNesting
                0x20000150        0x4 C:\Eclipse\Firmware\FreeRTOS\SAM4E\libFreeRTOS.a(port.o)



200048dc	 .bss.Wire      0x200048dc       0x20 ./src/Hardware/SAM4E/Devices.o


200049a5	 .bss._ZN6IoPort15logicalPinModesE
                0x200049a5       0xa6 ./src/Hardware/IoPorts.o
                0x200049a5                IoPort::logicalPinModes

I'm lost because the restarts are random and occur after a while, not at the first execution of a particular macro. I'm pretty sure that the error occured during filament loading, in the middle of a slow move to an endstop: G1 H4 V25 F300 in the macro 0:/sys/MMU Control/loadToBondtech.g

I've read all the topics about this error code and reason so I checked all the power terminals and they are are alright and the WIFi module knows about only one SSID..

Any ideas?

Leonard03

As an update, now I've completed the same gcode that in the first post failed. Now the exact file has completed successfully in 7h 49m 40s

The only difference was to disable the write buffer via M122 P500 S0
The diagnostics after this print job is this:

10/20/2023, 7:07:53 PM: M122: === Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.5.0-rc.1+ (2023-09-27 13:34:46) running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-917NK-F2MS4-7JKDG-3S06M-9ZSWD
Used output buffers: 4 of 26 (26 max)
=== RTOS ===
Static ram: 23076
Dynamic ram: 79228 of which 52 recycled
Never used RAM 7140, free system stack 110 words
Tasks: NETWORK(2,nWait,191.2%,182) HEAT(3,nWait,1.2%,308) Move(4,nWait,20.6%,261) DUEX(5,nWait,0.0%,26) MAIN(1,running,156.4%,691) IDLE(0,ready,0.6%,29), total 370.1%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 08:12:17 ago, cause: software
Last software reset at 2023-10-20 10:55, reason: User, Gcodes spinning, available RAM 7260, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x0c
Aux0 errors 0,0,0
MCU temperature: min 25.2, current 25.8, max 27.0
Supply voltage: min 11.8, current 12.3, max 12.4, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/23, heap memory allocated/used/recyclable 2048/588/128, gc cycles 138
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0
Driver 1: standstill, SG min 0
Driver 2: standstill, SG min 0
Driver 3: standstill, SG min 0
Driver 4: standstill, SG min 0
Driver 5: standstill, SG min 0
Driver 6: standstill, SG min 0
Driver 7: standstill, SG min 0
Driver 8: standstill, SG min n/a
Driver 9: standstill, SG min n/a
Driver 10: 
Driver 11: 
Date/time: 2023-10-20 19:07:50
Cache data hit count 4294967295
Slowest loop: 215.79ms; fastest: 0.14ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 47.5ms, write time 1.6ms, max retries 0
=== Move ===
DMs created 83, segments created 33, maxWait 12407ms, bed compensation in use: mesh, height map offset 0.009, ebfmin -1.00, ebfmax 1.00
no step interrupt scheduled
Moves shaped first try 15140, on retry 2857, too short 20305, wrong shape 63713, maybepossible 4509
=== DDARing 0 ===
Scheduled moves 294957, completed 294957, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Q0 segments left 0
Code queue 0 is empty
=== DueX ===
Read count 1804, 4.40 reads/min
=== Network ===
Slowest loop: 34.03ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta5
MAC address b4:e6:2d:52:f5:47
Module reset reason: Power up, Vcc 3.40, flash size 2097152, free heap 43056
WiFi IP address 192.168.0.107
Signal strength -49dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0
10/20/2023, 7:08:00 PM: M122 P500: Write buffer is disabled

dc42

@Leonard03 when we see an invState hard fault, the stack trace is generally nonsensical, and substantially different every time the hard fault occurs. We suspect that most or all of these types of faults are caused by transients or static discharge. One source pf static discharge is between the hot end metalwork and the thermistor connections, because the act of extruding plastic causes static charge to build up. Therefore, please ensure that your hot end metalwork is connected to Duet ground, either directly or through a resistor of e.g. 100Kohms.
If these hard faults continue to occur, please post the M122 reports afterwards so that we can evaluate whether there are any common factors in the stack traces.

Leonard03

@dc42 thank you for the answer. Sadly I can't find a good way to ground the V6 heatsink from the cold side. It is possible to use TVS diodes as a workaround? I'm using a PT100 daughterboard. It is possible to add at least one diode to one of the temperature sensor? Or they will affect the sensor/daughterboard functionality?

dc42

@Leonard03 please install the latest 3.5rc1+ firmware from https://www.dropbox.com/scl/fo/tjznycpk7bv7sj71p0ssl/h?rlkey=096p4nvgmigyrb20jj8olg3wu&dl=0, and post the M122 report next time this happens.

Do you still have a copy of the .map file for the version of RRF that generated the M122 report in your original post?

Leonard03

@dc42 Thank you very much for your response and for the binaries.
I will download them now and test tomorrow and report back

@dc42 said in Duet 2 WiFi HardFault invState:

Do you still have a copy of the .map file for the version of RRF that generated the M122 report in your original post?

Yes, I attached it here, but I changed the extension jut to have the ability to add it here. The extension is .map not .binDuet2CombinedFirmware_MAP.bin

Leonard03

@dc42 It works! And it works flawlessly!
With this build I've done some multihour prints with many tool changes and they worked perfect, no more resets (at least from thar reason).
For the PanelDue, using PanelDueFirmware-3.5.0-rc7-testcomm DWC is more responsive, more fluid, it updates more often and the connection is more consistent.
As a side note - annoying but don't bother me too much - after some time the DWC disappears altogether. I can ping the board, but no DWC. But until that point works flawlessly. By the way, trying to disable and reenable the wifi module (idle or printing) resulting in a stuckinspinloop when trying to enable it again.

Again, thank you very much for your time and help solving this issue!