Reboots/crashes - RRF ≤3.5.0-rc1

[Exerqtor]

And more:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1+ (2023-12-07 09:49:06) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 123248 of which 0 recycled
Never used RAM 12468, free system stack 180 words
Tasks: NETWORK(1,ready,38.7%,232) HEAT(3,nWait,0.2%,328) Move(4,nWait,0.0%,343) CanReceiv(6,nWait,0.4%,774) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.1%,350) TMC(4,nWait,6.2%,108) MAIN(1,running,2.9%,670) IDLE(0,ready,43.9%,29) AIN(4,delaying,7.6%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 08:58:00 ago, cause: software
Last software reset at 2023-12-09 01:46, reason: AssertionFailed, Platform spinning, available RAM 11580, slot 2
Software reset code 0x4920 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000913 000af408 0002de51 2002bed4 2002be01 000001a7 2002c100 20030440 2002c118 2001e888 a5a5a5a5 a5a5a5a5 0d362e30 00000000 00000000 00000000 2003044c 00000800 20035988 2002c100 20018678 2002bf9d 20018678 2001e888 000300db 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 32280441, completed 32280439, timed out 0, errs 0
MCU temperature: min 35.7, current 36.6, max 39.1
Supply voltage: min 22.3, current 24.1, max 26.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/26, heap memory allocated/used/recyclable 2048/1936/1596, gc cycles 64514
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 1, ifcnt 63, reads 60543, writes 13, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x71
Driver 1: standstill, SG min 228, read errors 0, write errors 1, ifcnt 62, reads 60541, writes 13, timeouts 3, DMA errors 0, CC errors 0, failedOp 0x6c
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 99, reads 60544, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 103, reads 60544, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 99, reads 60545, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-09 10:44:02
Cache data hit count 4294967295
Slowest loop: 12.87ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.8ms, write time 4.4ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 290534, received 661804, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 344, free buffers 26 (min 25), ts 161403/161402/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 5.80ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 3 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 39348
WiFi IP address 192.168.10.x
Signal strength -48dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0 

[dc42]

@Exerqtor thanks. The text that is overwriting the return address s now "0.6<CR>". The fact that is includes a <CR> suggests that this text comes from a job file or macro file, not from RRF itself, because RRF uses <LF> as a line terminator and almost never generates <CR>.

Please can you insert command G4 S10000 at the start of daemon.g and restart, to see if it still crashes.

[dc42]

@Exerqtor which browser are you using, and what version? That text ""0.6<CR>" could be the end of a browser version in a HTTP header.

PS - also, what language settings do you have in your browser? That text could be the end of an Accept-Language HTTP header.

[dc42]

@Exerqtor one more data point please: have you always been running a 2.1beta version of DuetWiFiServer when you had these errors, or did you have them when using version 1.27 also?

[Exerqtor]

@dc42
Ok let's see if I'm able to answer all that somewhat effectively 🤣

---

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor thanks. The text that is overwriting the return address s now "0.6<CR>". The fact that is includes a <CR> suggests that this text comes from a job file or macro file, not from RRF itself, because RRF uses <LF> as a line terminator and almost never generates <CR>.

Please can you insert command G4 S10000 at the start of daemon.g and restart, to see if it still crashes.

First off, is this what you had in mind?:

; /sys/daemon.g  v2.5
; Used to execute regular tasks, the firmware executes it and once the end of file is reached it waits. If the file is not found it waits and then looks for it again.

;---/
; -/--/--/--/--/--/--/--/--/--/--/--/--/--/--/--/--
; THIS MACRO ONLY WORKS WITH RRF 3.5.0b1 AND LATER!!
;--/--/--/--/--/--/--/--/--/--/--/--/--/--/--/--/--
;-/

G4 S10000                                                                      ; As per David's instructions

; Loop, to be able to turn on/off daemon.g
while global.RunDaemon
  ; Stuff goes below this line
  ; ---------------------------------------------------------------------------         

  ; Resume after filament change
  if exists(global.FilamentCHG)
    if global.FilamentCHG = true && state.status="paused"
      M24                                                                      ; Resume the pause automatically now that the manual filament change is done
      set global.FilamentCHG = false

  ; --------------------

  ; Refresh chamber lights status
  if exists(global.chamber_leds)
    set global.chamber_leds = state.gpOut[0].pwm * 100

  ; --------------------

  ;Check sb_leds status
  var SB_LEDS = true                                                           ; Turn on(true) / off(false) the Stealthburner led "system"
  if var.SB_LEDS
    if fileexists("/sys/lib/led/sb_leds-state.g")
      M98 P"/sys/lib/led/sb_leds-state.g"                                      ; Check if global.sb_leds has changed since last run/loop

  ; ---------------------------------------------------------------------------
  ; Daemon loop delay
  G4 P10                                                                        ; Delay running again or next command for at least 1 second (was S1)

;abcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcd
;abcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcd
;abcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcd
;abcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcd
;abcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcdbcdabcdabcdabcd

Or would you want it within the "RunDaemon" loop, and remove/comment out the one in the bottom @dc42 ?

---

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor which browser are you using, and what version? That text ""0.6<CR>" could be the end of a browser version in a HTTP header.

PS - also, what language settings do you have in your browser? That text could be the end of an Accept-Language HTTP header.

This is somewhat difficult to answer since i have no idea what Orca Slicer v1.8.1 (English language settings) runs under the hood edit Orca Slicer is running WebView2 /edit (and I have the "Device" tab open within Orca Slicer). For my desktop browser i'm currently using "Chrome x64 version 119.0.6045.200"(Norwegian language settings), but that's being updated all the time.

---

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor one more data point please: have you always been running a 2.1beta version of DuetWiFiServer when you had these errors, or did you have them when using version 1.27 also?

I'm quite positive that I tried several DuetWiFiServer releases once i started having these issues, trying to pinpoint where they came from. But since the crashes also kept happening on 1.27 i updated to the newest 2.1 beta again (if my memory serves right).

---

Other than that it had another crash now while i was about to deploy the changes to daemon.g, so i'll post that first:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1+ (2023-12-07 09:49:06) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 123200 of which 0 recycled
Never used RAM 12516, free system stack 182 words
Tasks: NETWORK(1,ready,19.9%,211) HEAT(3,nWait,0.0%,353) Move(4,nWait,0.0%,357) CanReceiv(6,nWait,0.0%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,0.7%,114) MAIN(1,running,71.7%,670) IDLE(0,ready,6.8%,29) AIN(4,delaying,0.9%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:08:56 ago, cause: software
Last software reset at 2023-12-09 11:38, reason: HardFault bfarValid precise, Platform spinning, available RAM 11420, slot 0
Software reset code 0x4060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00000803 BFAR 0x0d362e38 SP 0x20011f48 Task NETW Freestk 466 ok
Stack: 0d362e30 20032cac 000001bf 00000000 0d362e30 20032cac 000043a8 010f0200 00000001 200117a4 2002bed0 000001bf 000001bf 00000004 0002dd33 2002bed4 2002be01 000001bf 2002c100 20031c70 2002c118 2001e888 362e303d a5a5a5a5 0002ff41 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 536827, completed 536827, timed out 0, errs 0
MCU temperature: min 36.8, current 36.8, max 37.2
Supply voltage: min 24.0, current 24.1, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/27, heap memory allocated/used/recyclable 2048/384/32, gc cycles 1064
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 16, read errors 0, write errors 0, ifcnt 86, reads 651, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 246, read errors 0, write errors 0, ifcnt 85, reads 651, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 117, reads 652, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 122, reads 651, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 117, reads 651, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-09 11:47:28
Cache data hit count 922088436
Slowest loop: 8.47ms; fastest: 0.15ms
=== Storage ===
Free file entries: 17
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.0ms, write time 4.1ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 112, received 251, lost 0, errs 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 3, free buffers 26 (min 26), ts 62/62/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 21.95ms; fastest: 0.07ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 43140
WiFi IP address 192.168.10.x
Signal strength -50dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

[Exerqtor]

I did a little change to daemon.g to avoid "hammering" RRF with a new global.chamber_leds value change every time it's ran, unless the value has changed from last time it was ran. Don't know if this will help this issue, but it should offload RRF a tiny bit at least:

  ; Refresh chamber lights status
  if exists(global.chamber_leds)
    if global.chamber_leds != {state.gpOut[0].pwm * 100}
      set global.chamber_leds = state.gpOut[0].pwm * 100

---

Another crash btw, it happened while the machine was idle but i pulled the report while printing (if that matters):

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1+ (2023-12-07 09:49:06) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 13 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 126540 of which 12 recycled
Never used RAM 7980, free system stack 136 words
Tasks: NETWORK(1,ready,18.2%,195) HEAT(3,nWait,0.0%,353) Move(4,nWait,0.0%,244) CanReceiv(6,nWait,0.1%,797) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,0.7%,74) MAIN(1,running,80.1%,580) IDLE(0,ready,0.0%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:59:00 ago, cause: software
Last software reset at 2023-12-09 15:05, reason: HardFault bfarValid precise, Gcodes spinning, available RAM 6040, slot 2
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00000803 BFAR 0x362e303d SP 0x20011f60 Task NETW Freestk 472 ok
Stack: 362e303d 20032cfd 00000047 362e303c 20032ebc 0002dd33 00004434 210f0000 0002dce5 2002bed4 2002be01 000001c0 2002c100 200314b0 2002c118 2001e888 a5a5a5a5 a5a5a5a5 0002ff41 00000000 00000000 00000000 362e303d 00000800 200359c8 2002c100 20018678
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 3541025, completed 3541025, timed out 0, errs 0
MCU temperature: min 35.9, current 40.2, max 41.2
Supply voltage: min 22.9, current 24.2, max 26.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 198/22, heap memory allocated/used/recyclable 2048/2008/1700, gc cycles 7
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 179, reads 55243, writes 31, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 177, reads 55243, writes 31, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 172, reads 55253, writes 20, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 176, reads 55252, writes 21, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 171, reads 55253, writes 21, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-09 16:04:56
Cache data hit count 4294967295
Slowest loop: 223.45ms; fastest: 0.09ms
=== Storage ===
Free file entries: 15
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 5.0ms, write time 4.6ms, max retries 0
=== Move ===
DMs created 83, segments created 16, maxWait 3438848ms, bed compensation in use: mesh, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 20, on retry 14, too short 0, wrong shape 7, maybepossible 0
=== DDARing 0 ===
Scheduled moves 129, completed 129, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.6
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is doing "M116 H1" in state(s) 0 0, running macro
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S10000" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0, sync state 1
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 31998, received 72814, lost 0, errs 1, boc 0
Longest wait 6ms for reply type 6013, peak Tx sync delay 265, free buffers 26 (min 25), ts 17705/17704/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 71.81ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42944
WiFi IP address 192.168.10.x
Signal strength -52dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

[dc42]

@Exerqtor please try the firmware binary at https://www.dropbox.com/scl/fo/uf8gsgetc2mfbh34hm3up/h?rlkey=bx1aavry6xplebwxhv5rx7xee&dl=0 and test it both for the WiFi resetting issue and the extruder reversal issue.

I've repeated this message in your other thread so that we can keep track of the two issues independently.

[Exerqtor]

@dc42 Ok let's see, I've installed those last binaries and the "normal" 3.5.0rc1+ release on the 1LC.

After that I tried one of the jobs that failed earlier, and it completed without reversing the extruder (or crashing).

I'll try some of the other jobs that has failed too now and see of they also complete.

Should i still leave the "stuff" we put in daemon.g be there for a while btw?

(Copying this response to the other thread too, since it's refering to the same binaries)

[dc42]

@Exerqtor I don't think the stuff in daemon.g is relevant.

Are you able to take a Wireshark trace of your browser talking to the Duet? I have a theory that the "0.6<CR>" may be from the Accept-Language header that Chrome is sending

[Exerqtor]

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor I don't think the stuff in daemon.g is relevant.

Are you able to take a Wireshark trace of your browser talking to the Duet? I have a theory that the "0.6<CR>" may be from the Accept-Language header that Chrome is sending

Okol, i'll remove the G4 before the loop in daemon.g, and keep the rest.

And I will try to look into the Wireshark trace. Have to read up on what it is first

But I do also think the issue is Chrome, the last 24 hours I have only had two instances of Orca Slicer with the "Device" tab open and it haven't crashed once. If i do the same thing with two chrome tabs open it would have.

[Exerqtor]

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor I don't think the stuff in daemon.g is relevant.

Are you able to take a Wireshark trace of your browser talking to the Duet? I have a theory that the "0.6<CR>" may be from the Accept-Language header that Chrome is sending

I THINK, I've managed to do that trace now. But i can't see "0.6<CR>" in any of the data I managed to grab, or should i keep a trace going until it crashes and track it down?

Here is a HTTP trace from Chrome:

Frame 456: 475 bytes on wire (3800 bits), 475 bytes captured (3800 bits) on interface \Device\NPF_{DF4300F0-1489-452B-B73B-3870E63C20C9}, id 0
Ethernet II, Src: LiteonTechno_0e:23:17 (c8:ff:28:0e:23:17), Dst: Espressif_ce:91:93 (c4:5b:be:ce:91:93)
Internet Protocol Version 4, Src: 192.168.10.xx, Dst: 192.168.10.x
Transmission Control Protocol, Src Port: 63171, Dst Port: 80, Seq: 1, Ack: 1, Len: 421
Hypertext Transfer Protocol
    GET /rr_model?flags=d99fno HTTP/1.1\r\n
    Host: 192.168.10.x\r\n
    Connection: keep-alive\r\n
    X-Session-Key: 1319375919\r\n
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\r\n
    Accept: */*\r\n
    Referer: http://192.168.10.x/\r\n
    Accept-Encoding: gzip, deflate\r\n
    Accept-Language: nb-NO,nb;q=0.9,no;q=0.8,nn;q=0.7,en-US;q=0.6,en;q=0.5,sv;q=0.4,da;q=0.3,de;q=0.2,pt;q=0.1\r\n
    \r\n
    [Full request URI: http://192.168.10.x/rr_model?flags=d99fno]
    [HTTP request 1/1]
    [Response in frame: 470]

And one from Orca Slicer:

Frame 457: 479 bytes on wire (3832 bits), 479 bytes captured (3832 bits) on interface \Device\NPF_{DF4300F0-1489-452B-B73B-3870E63C20C9}, id 0
Ethernet II, Src: LiteonTechno_0e:23:17 (c8:ff:28:0e:23:17), Dst: Espressif_ce:91:93 (c4:5b:be:ce:91:93)
Internet Protocol Version 4, Src: 192.168.10.xx, Dst: 192.168.10.x
Transmission Control Protocol, Src Port: 63170, Dst Port: 80, Seq: 1, Ack: 1, Len: 425
Hypertext Transfer Protocol
    GET /rr_model?flags=d99fno HTTP/1.1\r\n
    Host: 192.168.10.x\r\n
    Connection: keep-alive\r\n
    X-Session-Key: 4102879531\r\n
    User-Agent: BBL-Slicer/v01.07.07.89 (dark) Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.52\r\n
    Accept: */*\r\n
    Referer: http://192.168.10.x/\r\n
    Accept-Encoding: gzip, deflate\r\n
    Accept-Language: nb,no;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6\r\n
    \r\n
    [Full request URI: http://192.168.10.x/rr_model?flags=d99fno]
    [HTTP request 1/1]
    [Response in frame: 466]

[NeoDue]

@Exerqtor sorry for chiming in once more...: I use Vivaldi here which is also Chromium-based, and I noticed that it works fine on my Desktop PC, but opened DWC tabs frequently crash on my notebook (either reloading endlessly or showing that "dead bird" symbol which is Vivaldi's replacement of the "Aw, snap" crash message of a Chromium tab) if the computer goes to some sleep or hibernation state (which the desktop PC does not do) and is resurrected later by moving the mouse or pressing a keyboard key.

With your finding here, I now see that as far as I can remember all the sudden crashes of my Duet which I blamed to "well, 3.5 is still in beta..." happened when the notebook was in use... don't know if that really means something though.

[Exerqtor]

@NeoDue Interesting! Well if it turns out that it's something chrome/chromium sends that's causing the crashes i guess it might explain it 😅

---

@dc42 I let one chrome and one Orca Slicer tab open over night with the wireshark trace going (so we maybe could see which of them sends the "CR") .

And when I woke up today I saw that the machine had crashed at least once, so I checked the debug log and it turns out it crashed 4 times from 20:30 until I got up at 5:45.

I've stoped and saved the trace data, but haven't had time to through it yet.

Here is a fresh M122 at least:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1++wdb (2023-12-11 09:30:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 122040 of which 0 recycled
Never used RAM 13676, free system stack 182 words
Tasks: NETWORK(1,ready,22.7%,239) HEAT(3,nWait,0.0%,331) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.0%,774) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.0%,350) TMC(4,nWait,0.7%,108) MAIN(1,running,70.5%,670) IDLE(0,ready,5.1%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 01:34:18 ago, cause: software
Last software reset at 2023-12-12 04:54, reason: AssertionFailed, Gcodes spinning, available RAM 13676, slot 0
Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00487000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000919 000af3d0 0002de6f 2002bed4 00000001 000001ae 2002c100 20031c70 2002c118 2001e888 a5a5a5a5 a5a5a5a5 0a0d312e 00000000 00000000 00000000 20031c7c 00000800 20035970 2002c100 20018678 2002bf9d 20018678 2001e888 0003010f 00000000 00000000
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 5659109, completed 5659109, timed out 0, errs 0
MCU temperature: min 34.4, current 34.7, max 37.6
Supply voltage: min 22.2, current 24.1, max 26.8, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/23, heap memory allocated/used/recyclable 2048/1364/1052, gc cycles 10814
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 18, read errors 0, write errors 1, ifcnt 20, reads 35673, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 18, reads 35673, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 157, reads 35672, writes 13, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x72
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 159, reads 35672, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 154, reads 35673, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-12 06:29:12
Cache data hit count 4294967295
Slowest loop: 13.56ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.7ms, write time 4.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P10" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 50942, received 116052, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6031, peak Tx sync delay 270, free buffers 26 (min 25), ts 28296/28295/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 6.33ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 3 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 36268
WiFi IP address 192.168.10.x
Signal strength -46dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

[dc42]

@Exerqtor when I say <CR> I mean the control character carriage return. Wireshark displays it as \r so OrcaSlicer does indeed send that sequence at the end of the Accept-Language header.

Do you get any crashes if you connect Chrome to the Duet but not OrcaSlicer?

[Exerqtor]

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor when I say <CR> I mean the control character carriage return. Wireshark displays it as \r so OrcaSlicer does indeed send that sequence at the end of the Accept-Language header.

Oh there you go (this is out of my depth so sorry for the missunderstanding 😅).

Do you get any crashes if you connect Chrome to the Duet but not OrcaSlicer?

I'll give it a try later today, remove the IP adress from Orca Slicer entirely to rule it out.

---

Just out of curiosity, if this turns out being the reason for the crashing. Is it a Duet 3 Mini WiFi specific issue, or does it apply to Duet 2 WiFi / Duet 3 6HC WiFi as well?

[NeoDue]

@Exerqtor I have a 3 6HC here. Thus, if my assumption from yesterday is correct this applies to that board as well. But I guess it might be heavily dependent on how Chromium works on the corresponding computer, including anything that might be affecting this, such as e.g. how the computer handles standby states.

I will only be able to test my assumption in about a week since I am not at home, but I will follow this thread and try to recreate that issue both on my Duet 3 6HC as well as on my old Duet 2 then unless you already found a solution by then - you two are working really fast at the moment

[Exerqtor]

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor when I say <CR> I mean the control character carriage return. Wireshark displays it as \r so OrcaSlicer does indeed send that sequence at the end of the Accept-Language header.

Do you get any crashes if you connect Chrome to the Duet but not OrcaSlicer?

The IP address is now removed from Orca Slicer, and I've got two DWC instances opened in chrome.

So lets play the waiting game and see what happens.

[Exerqtor]

@dc42 said in Reboots/crashes - RRF 3.5.0-rc1:

@Exerqtor when I say <CR> I mean the control character carriage return. Wireshark displays it as \r so OrcaSlicer does indeed send that sequence at the end of the Accept-Language header.

Do you get any crashes if you connect Chrome to the Duet but not OrcaSlicer?

It's not Orca Slicer or "just" Orca Slicer thats causing the crashes, it happened again with it closed/not connected:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1++wdb (2023-12-11 09:30:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 2 of 40 (39 max)
=== RTOS ===
Static ram: 102844
Dynamic ram: 123224 of which 208 recycled
Never used RAM 12284, free system stack 180 words
Tasks: NETWORK(2,nWait,193.2%,215) HEAT(3,nWait,0.2%,327) Move(4,nWait,0.0%,352) CanReceiv(6,nWait,0.4%,774) CanSender(5,nWait,0.0%,336) CanClock(7,delaying,0.1%,350) TMC(4,nWait,6.4%,108) MAIN(1,running,250.4%,670) IDLE(0,ready,42.9%,29) AIN(4,delaying,7.6%,264), total 501.1%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 03:34:40 ago, cause: software
Last software reset at 2023-12-13 02:36, reason: AssertionFailed, Gcodes spinning, available RAM 11124, slot 1
Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011fbc Task NETW Freestk 495 ok
Stack: 00000919 000af3d0 0002de6f 2002bed4 2002be01 000001af 2002c100 20031458 2002c118 2001e888 0d0a0d31 a5a5a5a5 0d312e30 00000000 00000000 00000000 20031464 00000800 20035970 2002c100 20018678 2002bf9d 20018678 2001e888 0003010f 00000000 00000000
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 12881155, completed 12881155, timed out 0, errs 0
MCU temperature: min 33.6, current 33.8, max 37.3
Supply voltage: min 21.9, current 24.0, max 27.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/27, heap memory allocated/used/recyclable 2048/708/356, gc cycles 26104
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 50, reads 22552, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 16, read errors 0, write errors 1, ifcnt 48, reads 22552, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 176, reads 22552, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 178, reads 22551, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 172, reads 22552, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-13 06:11:01
Cache data hit count 4294967295
Slowest loop: 12.69ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.9ms, write time 4.5ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P10" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
in 0 notIn 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 115938, received 264103, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 276, free buffers 26 (min 25), ts 64405/64404/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 6.29ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 3 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 39692
WiFi IP address 192.168.10.x
Signal strength -47dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Doing the same with Orca Slicer now, no chrome connections and two Orca Slicer.

[NeoDue]

@Exerqtor may I ask about the computer you ran Chrome on in that test? Which OS? Notebook or Desktop? Screensaver, sleep or hibernation / energy saving mode enabled or disabled? Unless I overlook something, one of these things probably made the difference here.

[timschneider]

@dc42
Hi david, I'm reading this thread as bystander. I asked myself if it is possible that the error occurs when the connection is interrupted but the socket is not closed and RRF is running in an timeout?

Maybe I'm totally off, but as far as I know the RRF is opening an DMA channel to the WiFi/Network stack and the socket handling is done over there.

What will happen if a client is sending a partially delayed http header like

GET /rr_model?flags=d99fno HTTP/1.1\r\n
    Host: 192.168.10.x\r\n
    Connection: keep-alive\r\n
    X-Session-Key: 4102879531\r\n
    User-Agent: BBL-Slicer/v01.07.07.89 (dark) Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.52\r\n
    Accept: */*\r\n
    Referer: http://192.168.10.x/\r\n
<VERY LONG DELAY>
    Accept-Encoding: gzip, deflate\r\n
    Accept-Language: nb,no;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6\r\n
\r\n

The socket will stay open and later on run into an timeout, but the DMA channel is still open and partially written. The buffer gets freed (on stack), but the client or WiFi/Network stack keeps writing to that address via the opened DMA after the <very long delay>.

Maybe the idea is totally off, then just ignore it.

I was just thinking about another behaviour I observed when using DWC over an VPN tunnel, sometimes the RRF just stopped after I checked the status of the print and disconnected the VPN.

Edit:

• No timeout in the Network stack
• Timeout in the RRF
• Open DMA channel