Reboots/crashes - RRF ≤3.5.0-rc1

[timschneider]

@dc42
Hi david, I'm reading this thread as bystander. I asked myself if it is possible that the error occurs when the connection is interrupted but the socket is not closed and RRF is running in an timeout?

Maybe I'm totally off, but as far as I know the RRF is opening an DMA channel to the WiFi/Network stack and the socket handling is done over there.

What will happen if a client is sending a partially delayed http header like

GET /rr_model?flags=d99fno HTTP/1.1\r\n
    Host: 192.168.10.x\r\n
    Connection: keep-alive\r\n
    X-Session-Key: 4102879531\r\n
    User-Agent: BBL-Slicer/v01.07.07.89 (dark) Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.52\r\n
    Accept: */*\r\n
    Referer: http://192.168.10.x/\r\n
<VERY LONG DELAY>
    Accept-Encoding: gzip, deflate\r\n
    Accept-Language: nb,no;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6\r\n
\r\n

The socket will stay open and later on run into an timeout, but the DMA channel is still open and partially written. The buffer gets freed (on stack), but the client or WiFi/Network stack keeps writing to that address via the opened DMA after the <very long delay>.

Maybe the idea is totally off, then just ignore it.

I was just thinking about another behaviour I observed when using DWC over an VPN tunnel, sometimes the RRF just stopped after I checked the status of the print and disconnected the VPN.

Edit:

• No timeout in the Network stack
• Timeout in the RRF
• Open DMA channel

[Exerqtor]

@NeoDue said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor may I ask about the computer you ran Chrome on in that test? Which OS? Notebook or Desktop? Screensaver, sleep or hibernation / energy saving mode enabled or disabled? Unless I overlook something, one of these things probably made the difference here.

It's on a desktop running windows 10 pro x64, hibernation/powersaving has been turned off since i'm running these tests.

So it's eiter chrome thats making the issues OR both. That we will see if it crashes with only Orca Slicer running (as it is now).

---

@dc42 So i just came home, and it's been no crashes since 05:50 this morning at least, this is with only Orca Slicer open. Gonna let it stay idle like this until tomorrow morning and see what happens. But i got a sneaking feeling Orca Slicer (by itself at least) ain't the cause of the crashes.

[dc42]

@Exerqtor have you ever seen this issue when running DuetWiFiServer 1.27? If you haven't, or you are not sure, please try it.

[Exerqtor]

@dc42 Not sure, but i'm pretty sure it came after moving from 1.27. I'll revert to it now, shouldn't be an issue to run 1.27 with 3.5x on everything else?

[T3P3Tony]

@Exerqtor you can use 1.27 with 3.5x

[Exerqtor]

Ok, with that it's now on WiFiServer 1.27, and i'm back to having two tabs with chrome and one Orca Slicer instance (since that seem to be what made it crash most often). #waitinggame

[Exerqtor]

@T3P3Tony & @dc42 Just had a crash on WiFiServer 1. 27 as well:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1++wdb (2023-12-11 09:30:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 123224 of which 0 recycled
Never used RAM 12492, free system stack 182 words
Tasks: NETWORK(1,ready,23.6%,219) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,0.7%,108) MAIN(1,running,69.2%,670) IDLE(0,ready,5.6%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:28:00 ago, cause: software
Last software reset at 2023-12-13 19:25, reason: AssertionFailed, Gcodes spinning, available RAM 10924, slot 2
Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000919 000af3d0 0002de6f 2002bed4 2002be01 000001af 2002c100 20030c40 2002c118 2001e888 a5a5a5a5 0d0a0d36 0d312e30 00000000 00000000 00000000 20030c4c 00000800 20035970 2002c100 20018678 2002bf9d 20018678 2001e888 0003010f 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1680921, completed 1680920, timed out 0, errs 0
MCU temperature: min 35.5, current 36.2, max 39.1
Supply voltage: min 11.5, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/27, heap memory allocated/used/recyclable 2048/404/52, gc cycles 3230
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 97, reads 22904, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 95, reads 22904, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 200, reads 22904, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 203, reads 22903, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 197, reads 22904, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-13 19:53:05
Cache data hit count 2936262464
Slowest loop: 10.26ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.6ms, write time 4.6ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P10" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 15138, received 34498, lost 0, errs 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 265, free buffers 26 (min 25), ts 8405/8404/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 13.65ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 4 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 1.27
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 25464
WiFi IP address 192.168.10.x
Signal strength -55dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

[NeoDue]

@Exerqtor thanks! Let me add my observations in detail, maybe they do help someone here - I honestly don't know what happens in detail when the browser is open on a "sleeping" computer:

1. Desktop PC running MX Linux or Linux Mint with Vivaldi open and no kind of powersaving except turning the monitor off active: dwc tabs do not crash, Duet does not crash.
2. Notebook (in my case an old MacBook Air) running Windows 10 x64 with the same Vivaldi version active, hibernation turned off, but higher sleep modes active: tabs with DWC frequently do not recover if the computer is woken from sleep. Duet crashes occur (on a Duet 3 6HC running RRF 3.5.0 beta 4 or 3.5.0rc1), but not every time the browser tab crashes. Several browser tab crashes seem to increase the risk for the Duet itself crashing however, as far as I remember.

I still have my old printer here which runs on a Duet 2 Wifi with RRF 3.3 or an early 3.4 version. If it helps anyone, I could try to recreate those crashes in the new board and then check if the old one might be affected as well.

[Exerqtor]

@NeoDue Interested in seeing if you're also able to recreate this on your setup with the same results!

---

Otherwise it's another crash over the night, same test criteria as the last report:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1++wdb (2023-12-11 09:30:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 8 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 122040 of which 0 recycled
Never used RAM 13676, free system stack 182 words
Tasks: NETWORK(2,nWait,410.5%,236) HEAT(3,nWait,0.4%,327) Move(4,nWait,0.0%,356) CanReceiv(6,nWait,0.9%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.1%,350) TMC(4,nWait,12.4%,108) MAIN(1,running,1216.0%,670) IDLE(0,ready,97.1%,29) AIN(4,delaying,14.7%,264), total 1752.3%
Owned mutexes:
=== Platform ===
Last reset 01:41:13 ago, cause: software
Last software reset at 2023-12-14 04:08, reason: AssertionFailed, Gcodes spinning, available RAM 12492, slot 1
Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000919 000af3d0 0002de6f 2002bed4 2002be01 000001ae 2002c100 20030c58 2002c118 2001e888 a5a5a5a5 a5a5a5a5 0a0d312e 00000000 00000000 00000000 20030c64 00000800 20035988 2002c100 20018678 2002bf9d 20018678 2001e888 0003010f 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 6073286, completed 6073286, timed out 0, errs 0
MCU temperature: min 34.7, current 35.3, max 38.4
Supply voltage: min 21.7, current 24.0, max 27.4, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/23, heap memory allocated/used/recyclable 2048/1496/1184, gc cycles 11249
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 1, ifcnt 135, reads 57475, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 208, read errors 0, write errors 1, ifcnt 133, reads 57475, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 233, reads 57475, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 235, reads 57474, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 229, reads 57475, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-14 05:49:51
Cache data hit count 4294967295
Slowest loop: 14.59ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.7ms, write time 4.4ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 54669, received 124545, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6031, peak Tx sync delay 271, free buffers 26 (min 25), ts 30367/30366/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 12.60ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 4 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 1.27
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 22544
WiFi IP address 192.168.10.x
Signal strength -56dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

[T3P3Tony]

@NeoDue thanks for this information. We need the M122 report(s) for when the Duet crashes. The DWC not recovering from sleep modes etc may be related but may also be a different issue so lets leave that aside for now and focus on what happens when RRF crashes.

[NeoDue]

@T3P3Tony @Exerqtor okay. Just to be sure: This means I let the Duet crash and get an M122 afterwards. Anything I need to set on the Duet in advance?

[T3P3Tony]

@NeoDue It does not hurt to grab a M122 before you start but nothing specifically in advance.

[Exerqtor]

Just got back home from work, to another crash on 1.27.

Besides crashing (seemingly just as often as the 2.4 betas), DWC loads sooooo damned slow with 1.27!

Anywho, another report:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1++wdb (2023-12-11 09:30:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 14 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 122016 of which 0 recycled
Never used RAM 13700, free system stack 182 words
Tasks: NETWORK(2,nWait,23.4%,211) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,0.7%,108) MAIN(1,running,69.4%,670) IDLE(0,ready,5.6%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:18:49 ago, cause: software
Last software reset at 2023-12-14 15:39, reason: AssertionFailed, Gcodes spinning, available RAM 13676, slot 1
Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00489000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000919 000af3d0 0002de6f 2002bed4 2002be01 000001af 2002c168 20031ca0 2002c180 2001e888 a5a5a5a5 0d0a0d36 0d312e30 00000000 00000000 00000000 20031cac 00000800 200359a0 2002c168 20018678 2002bf9d 20018678 2001e888 0003010f 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1129622, completed 1129621, timed out 0, errs 0
MCU temperature: min 34.8, current 35.3, max 38.2
Supply voltage: min 21.7, current 24.1, max 27.4, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/22, heap memory allocated/used/recyclable 2048/1604/1304, gc cycles 2081
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 1, ifcnt 177, reads 59429, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 208, read errors 0, write errors 1, ifcnt 175, reads 59429, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 19, reads 59429, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 21, reads 59428, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 15, reads 59429, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-14 15:57:58
Cache data hit count 1957951974
Slowest loop: 13.50ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.8ms, write time 4.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 10177, received 23198, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6031, peak Tx sync delay 261, free buffers 26 (min 25), ts 5649/5648/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 12.58ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 4 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 1.27
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 25624
WiFi IP address 192.168.10.x
Signal strength -54dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

[timschneider]

@dc42
I was looking at the code and found your ocmment about the failed assert.

// Look at the response
#if SAME5x	//TEMP DEBUG
	CheckStackValue(9, ra);			//***** This is the check that occasionally fails ******
#endif

looking backwards from here it will reset the spi and disable the dma channel. There is a comment about the disabling of the dma channel on SAME5x.

// Disable a channel. Also clears its status and disables its interrupts.
// On the SAME5x it is sometimes impossible to disable a channel. So we now return true if disabling it succeeded, false it it is still enabled.
bool DmacManager::DisableChannel(const uint8_t channel) noexcept
{

but the return value is not checked in the function:

static inline void spi_rx_dma_disable() noexcept

From here I've looked in the datasheet of the chip to check the dma function and found the following:

So it is possible that a DMA transfer is already scheduled but not processed and this will block/prevent the gracefull disable from taking place.

[Exerqtor]

@timschneider said in Reboots/crashes - RRF 3.5.0-rc1:

@dc42
I was looking at the code and found your ocmment about the failed assert.

// Look at the response
#if SAME5x	//TEMP DEBUG
	CheckStackValue(9, ra);			//***** This is the check that occasionally fails ******
#endif

looking backwards from here it will reset the spi and disable the dma channel. There is a comment about the disabling of the dma channel on SAME5x.

// Disable a channel. Also clears its status and disables its interrupts.
// On the SAME5x it is sometimes impossible to disable a channel. So we now return true if disabling it succeeded, false it it is still enabled.
bool DmacManager::DisableChannel(const uint8_t channel) noexcept
{

but the return value is not checked in the function:

static inline void spi_rx_dma_disable() noexcept

From here I've looked in the datasheet of the chip to check the dma function and found the following:

So it is possible that a DMA transfer is already scheduled but not processed and this will block/prevent the gracefull disable from taking place.

Woah, great catch! Hope you're on to something there!

---

And some more data points:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1++wdb (2023-12-11 09:30:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 122016 of which 0 recycled
Never used RAM 13700, free system stack 186 words
Tasks: NETWORK(1,ready,23.2%,217) HEAT(3,nWait,0.0%,335) Move(4,nWait,0.0%,344) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,delaying,0.7%,108) MAIN(1,running,69.6%,670) IDLE(0,ready,5.5%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:25:42 ago, cause: software
Last software reset at 2023-12-14 20:16, reason: AssertionFailed, Expansion spinning, available RAM 13676, slot 2
Software reset code 0x4132 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000919 000af3d0 0002de6f 2002bed4 2002be01 000001af 2002c100 20031458 2002c118 2001e888 a5a5a5a5 a5a5a5a5 0d312e30 00000000 00000000 00000000 20031464 00000800 20035970 2002c100 20018678 2002bf9d 20018678 2001e888 0003010f 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1542150, completed 1542150, timed out 0, errs 0
MCU temperature: min 35.3, current 35.9, max 38.6
Supply voltage: min 21.9, current 24.1, max 27.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/23, heap memory allocated/used/recyclable 2048/2048/1736, gc cycles 2851
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 1, ifcnt 191, reads 15599, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 208, read errors 0, write errors 1, ifcnt 189, reads 15599, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 33, reads 15598, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 15598, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 29, reads 15599, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-14 20:42:33
Cache data hit count 2670623150
Slowest loop: 11.30ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 8.4ms, write time 3.6ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P10" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 13889, received 31654, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6031, peak Tx sync delay 300, free buffers 26 (min 25), ts 7711/7710/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 13.57ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 4 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 1.27
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 25624
WiFi IP address 192.168.10.x
Signal strength -53dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

[NeoDue]

@T3P3Tony thanks! Test is running now. One thing I found due to the browser updating itself soon after start: when the browser was closed for update with Duet tabs open, there is a brief "incomplete transfer" message on the PanelDue - and then the Duet resets the wifi connection (3.5.0rc1 with wifi module firmware 2.1b6 - at least reconnection seems reliable now with that firmware version). After that, one of the Duet tabs was dead. The only thing that differed in the log was the amount of reconnections.

I will ignore that for now as you suggested and wait for a crash to happen. The printer is set to write an m122 log into a file afer every reboot - let's see what I find tomorrow.

[Exerqtor]

@NeoDue
Looking forrward to seeing how this turns out for you, and if we have the same root cause for the crashes!

---

Other than that i'm moving over to WiFiServer 2.1b4 again, 1.27 is pretty much useless at this point. It flip out on reboot/crash and won't connect to the LAN with the only means i've found to connect it when that happens is to power toggle the printer til it manages to connect. And once it's finally able to connect, connecting to the printer/DWC is a total shitstorm, 90% of the times it just yields a white/blank screen. And once you're finally able to get a connection it's soooooo slow.

Here is the last crash report from 1.27 and a snipet from the debug log so you can see how it behaves when not being able to connect to the LAN (attached as a mega.nz file since it's a shit ton of data).

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1++wdb (2023-12-11 09:30:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 122100 of which 0 recycled
Never used RAM 13616, free system stack 186 words
Tasks: NETWORK(1,ready,23.6%,203) HEAT(3,nWait,0.0%,354) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,341) TMC(4,nWait,0.7%,114) MAIN(1,running,69.1%,670) IDLE(0,ready,5.7%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:07:20 ago, cause: power up
Last software reset at 2023-12-14 23:02, reason: AssertionFailed, none spinning, available RAM 13700, slot 0
Software reset code 0x4133 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000919 000af3d0 0002de6f 2002bed4 2002be01 000001af 2002c100 20031488 2002c118 2001e888 a5a5a5a5 a5a5a5a5 0d312e30 00000000 00000000 00000000 20031494 00000800 200359a0 2002c100 20018678 2002bf9d 20018678 2001e888 0003010f 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 440375, completed 440375, timed out 0, errs 0
MCU temperature: min 34.0, current 35.3, max 37.1
Supply voltage: min 0.9, current 24.0, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/23, heap memory allocated/used/recyclable 2048/1516/1204, gc cycles 801
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 23148, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 23148, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 23148, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 23147, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 23148, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-15 07:56:22
Cache data hit count 764831191
Slowest loop: 10.98ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.8ms, write time 4.4ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P10" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 3974, received 9009, lost 0, errs 1638, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 11121, free buffers 26 (min 25), ts 2202/2200/0
Tx timeouts 0,0,1,0,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 14.24ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 3 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 1.27
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 25536
WiFi IP address 192.168.10.x
Signal strength -54dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

The debug log file.

[NeoDue]

@Exerqtor so far no crashes here. They happened less often than they seemed to do at your place, so I will leave the test running.

I would however suggest you try to switch to Wifi firmware 2.1b6 instead of b4 - apart from debugging my config and macro files, that was the only thing I did with my printer since RC1 came out, and at least for me, that one solved some issues (albeit seemingly unrelated ones, but who knows...)

[Exerqtor]

@NeoDue said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor so far no crashes here. They happened less often than they seemed to do at your place, so I will leave the test running.

That's good to hear, on your behalf i mean

I would however suggest you try to switch to Wifi firmware 2.1b6 instead of b4 - apart from debugging my config and macro files, that was the only thing I did with my printer since RC1 came out, and at least for me, that one solved some issues (albeit seemingly unrelated ones, but who knows...)

Oh is there a 2.1b6? And where is that bugger hiding out?

---

Otherwise it's been two crashes while on 2.1b4, and here is a report:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1++wdb (2023-12-11 09:30:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 122016 of which 0 recycled
Never used RAM 13700, free system stack 180 words
Tasks: NETWORK(1,ready,57.1%,228) HEAT(3,nWait,0.1%,335) Move(4,nWait,0.0%,344) CanReceiv(6,nWait,0.1%,774) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,1.8%,108) MAIN(1,running,25.2%,670) IDLE(0,ready,13.5%,29) AIN(4,delaying,2.1%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 02:37:51 ago, cause: software
Last software reset at 2023-12-15 11:37, reason: AssertionFailed, none spinning, available RAM 12492, slot 2
Software reset code 0x4133 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000919 000af3d0 0002de6f 2002bed4 2002be01 000001b0 2002c100 20031c88 2002c118 2001e888 a5a5a5a5 a5a5a5a5 312e303d 00000000 00000000 00000000 20031c94 00000800 20035988 2002c100 20018678 2002bf9d 20018678 2001e888 0003010f 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 9471453, completed 9471453, timed out 0, errs 0
MCU temperature: min 34.7, current 35.3, max 37.8
Supply voltage: min 22.5, current 24.1, max 26.5, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/22, heap memory allocated/used/recyclable 2048/1376/1076, gc cycles 17933
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 1, ifcnt 58, reads 39703, writes 13, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x72
Driver 1: standstill, SG min 16, read errors 0, write errors 1, ifcnt 58, reads 39704, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 46, reads 39704, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 47, reads 39703, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 46, reads 39704, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-15 14:15:14
Cache data hit count 4294967295
Slowest loop: 13.44ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.7ms, write time 4.4ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 85250, received 194199, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 273, free buffers 26 (min 25), ts 47356/47355/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 7.31ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 3 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42948
WiFi IP address 192.168.10.x
Signal strength -52dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

[dc42]

@NeoDue said in Reboots/crashes - RRF 3.5.0-rc1:

Duet crashes occur (on a Duet 3 6HC running RRF 3.5.0 beta 4 or 3.5.0rc1)

Is this when using the optional WiFi module for the 6HC, or using Ethernet? Do you have any M122 reports from the 6HC taken after the Duet has reset?