Reboots/resets randomly - RRF 3.5.0-b4

Exerqtor

@dc42
Will do, just installed the new binaries. This is just meant to remedy the restart issue right? Not the reversing extruder?

dc42

@Exerqtor this may not remedy the resetting issue, but should remedy any extrusion issues.

Exerqtor

@dc42
Oh thats promising

dc42

@Exerqtor I just remembered you are using a tool board, so I've now put the latest tool board firmware build in the same place.

Exerqtor

@dc42
Happy to say i'm able to melt some plastic for the first time in almost two weeks now with that new build!

Looks like it fixed the extrusion issues!

dc42

@Exerqtor thanks, let me know ASAP if you get another main board reboot (and provide the M122 report) because I didn't really get to the bottom of that one.

Exerqtor

@dc42

Yeah I will shout out as soon as I see a reset in the debuglog!

Not any extrusion issues, but the print finishes off without completing the print for some reason. It's got status : "Cancelling" with 100% progress. And the printer is somewhat locked up in that state until i manually press Cancel in the status tab of DWC or PD (even though it's ran stop.g.

This is the last part of the print that just "finished":

G1 X176.576 Y189.748 E.00789
G1 X176.726 Y190.157 E.01028
G1 X176.791 Y190.611 E.01082
G1 X176.764 Y190.87 E.00614
G1 X176.969 Y191.069 F12000
; stop printing object print_job.stl id:0 copy 0
G10 ; retract

;TYPE:Custom
set global.line_type = "Custom"
; Filament-specific end G-code START
;
; Filament specific end G-code: END
; End G-code: START
M0
; End G-code: END
M73 P100 R0
; filament used [mm] = 2401.91
; filament used [cm3] = 5.78
; filament used [g] = 6.01
; filament cost = 1.68
; total filament used [g] = 6.01
; total filament cost = 1.68
; estimated printing time (normal mode) = 1h 23m 7s
; estimated first layer printing time (normal mode) = 2m 51s

; prusaslicer_config = begin
; autoemit_temperature_commands = 0
; avoid_crossing_curled_overhangs = 0
; avoid_crossing_perimeters = 0
; avoid_crossing_perimeters_max_detour = 0

And the debuglog:

2023-06-04 21:42:51 [debug] Done printing file
2023-06-04 21:42:51 [warn] Finished printing file 0:/gcodes/print_job_0.16mm_ABS_0.4n_1h23m.gcode, print time was 1h 12m
2023-06-04 21:42:56 [info] G10 P0 R0 S0                                                                   
2023-06-04 21:42:56 [debug] ABS filament unloaded & pressure advance disabled
2023-06-04 21:42:56 [debug] ABS filament unloaded & pressure advance disabled

jay_s_uk

@Exerqtor that appears to be if M0 is used at all then a print gets stuck. Try having stop.g but not using M0 to call it

Exerqtor

@jay_s_uk
Oh ok, I'll try that next time. I just started another print but disabled emitting M73's to see if that maybe was the issue.

Exerqtor

@dc42
New reset yesterday evening while idle:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.3+ (2023-05-24 17:47:02) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 3 of 40 (30 max)
=== RTOS ===
Static ram: 101720
Dynamic ram: 124184 of which 12 recycled
Never used RAM 13284, free system stack 182 words
Tasks: NETWORK(nWait,218.1%,219) HEAT(nWait,0.3%,327) Move(nWait,0.0%,358) CanReceiv(nWait,0.6%,662) CanSender(nWait,0.0%,337) CanClock(delaying,0.1%,342) TMC(nWait,14.4%,108) MAIN(running,134.0%,442) IDLE(ready,4.9%,29) AIN(delaying,10.1%,266), total 382.5%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 06:55:33 ago, cause: software
Last software reset at 2023-06-04 23:50, reason: HardFault bfarValid precise, Gcodes spinning, available RAM 11752, slot 0
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00000803 BFAR 0x00000004 SP 0x20011b98 Task NETW Freestk 482 ok
Stack: 0000019f 00000002 200014e8 0000019e 20032ecb 0009bebd 0002f872 810f0000 0002f869 00000000 00000000 00000000 20031cfc 00000800 200359f0 2002c058 20018260 2002bef5 20018260 2001e428 0002fa03 00000000 00000000 00000000 20011c48 00000014 b5dd9f97
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 24933600, completed 24933600, timed out 0, errs 0
MCU temperature: min 34.3, current 35.1, max 42.6
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/28, heap memory allocated/used/recyclable 2048/1364/980, gc cycles 932
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 249, reads 38425, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 249, reads 38425, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 99, reads 38425, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 100, reads 38424, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 98, reads 38425, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 197, reads 38428, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 197, reads 38428, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-06-05 06:45:47
Cache data hit count 4294967295
Slowest loop: 20.35ms; fastest: 0.14ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.4ms, write time 3.5ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1                                                                        " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 224410, received 498718, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 276, free buffers 18 (min 17), ts 124668/124667/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 20.32ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42988
WiFi IP address 192.168.10.50
Signal strength -61dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

dc42

@Exerqtor thanks.
~~Please install the new version at https://www.dropbox.com/sh/dxu8psema8yxpda/AACn_z2dkJVTAi16MHDH80HZa?dl=0 and send me the M122 report when it resets again.~~

Scratch that, the compiler optimised away the additional checks that I added.

dc42

@Exerqtor please install the new version at https://www.dropbox.com/sh/dxu8psema8yxpda/AACn_z2dkJVTAi16MHDH80HZa?dl=0 and send a M122 report when it next resets unexpectedly.

Exerqtor

@dc42
Todays first reset M122

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.3+ (2023-06-05 14:42:23) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: FAP55-Q967A-G65J0-401GL-0S02Z-RF9TL
Used output buffers: 1 of 40 (30 max)
=== RTOS ===
Static ram: 101940
Dynamic ram: 124180 of which 12 recycled
Never used RAM 13068, free system stack 182 words
Tasks: NETWORK(1,ready,167.8%,211) HEAT(3,nWait,0.2%,327) Move(4,nWait,0.0%,352) CanReceiv(6,nWait,0.4%,662) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.1%,351) TMC(4,nWait,11.1%,108) MAIN(1,running,113.9%,442) IDLE(0,ready,3.6%,29) AIN(4,delaying,7.5%,266), total 304.7%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 07:08:23 ago, cause: software
Last software reset at 2023-06-05 23:36, reason: AssertionFailed, Platform spinning, available RAM 11636, slot 1
Software reset code 0x4120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x20011bec Task NETW Freestk 503 ok
Stack: 00000139 000ace48 0002f52d 00000000 0002f8a9 00000000 00000000 00000000 20031d54 00000800 20035a48 2002c138 20018260 2002bfd5 20018260 2001e500 0002fa47 00000000 00000000 00000000 20011c48 00000014 00000000 00000002 d07a0050 640aa8c0 0800019e
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 25703859, completed 25703859, timed out 0, errs 0
MCU temperature: min 35.5, current 35.9, max 42.1
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/28, heap memory allocated/used/recyclable 2048/916/532, gc cycles 961
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 119, reads 42919, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 2, read errors 0, write errors 1, ifcnt 119, reads 42919, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 1, ifcnt 199, reads 42919, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 200, reads 42919, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 198, reads 42920, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 252, reads 42923, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 252, reads 42922, writes 10, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-06-06 06:44:52
Cache data hit count 4294967295
Slowest loop: 20.42ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.3ms, write time 2.1ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1                                                                        " in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 231343, received 514122, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 274, free buffers 18 (min 17), ts 128520/128519/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 20.40ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address bc:ff:4d:e6:b1:62
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 43000
WiFi IP address 192.168.10.50
Signal strength -53dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

dc42

@Exerqtor thanks, that's helped me to pin it down some more.

When the machine is idle, is anything on the Duet running apart from the web server responding to DWC, for example something in daemon.g, or an FTP client?

Exerqtor

@dc42 said in Extruder stops extruding (again) & reboots randomly - RRF 3.5b3+:

@Exerqtor thanks, that's helped me to pin it down some more.

Glad to hear!

When the machine is idle, is anything on the Duet running apart from the web server responding to DWC, for example something in daemon.g, or an FTP client?

Hmm no it shouldn't be as far as i'm aware? I added a "stopper" for the neopixel commands getting hammered every time daemon.g was ran before newyears so that shouldn't do anything atlesst.

I can post my current daemon.g later incase it's something there that might trigger the issue.

EDIT:

Just grabbed daemon.g from the printer (global.accel_control = false btw):

; /sys/daemon.g  v2.4
; Used to execute regular tasks, the firmware executes it and once the end of file is reached it waits. If the file is not found it waits and then looks for it again.

;---/
; -/--/--/--/--/--/--/--/--/--/--/--/--/--/--/--/--
; THIS MACRO ONLY WORKS WITH RRF 3.5.0b1 AND LATER!!
;--/--/--/--/--/--/--/--/--/--/--/--/--/--/--/--/--
;-/

; Loop, to be able to turn on/off daemon.g
while global.RunDaemon
  ; Stuff goes here         

  ; Resume after filament change
  if exists(global.FilamentCHG)
    if global.FilamentCHG = true && state.status="paused"
      M24                                                                      ; Resume the pause automatically now that the manual filament change is done
      set global.FilamentCHG = false

  ; Refresh chamber lights status
  if exists(global.chamber_leds)
    set global.chamber_leds = state.gpOut[0].pwm * 100

  ;Check sb_leds status
  ;if fileexists("/sys/lib/led/sb_leds-state.g")
    M98 P"/sys/lib/led/sb_leds-state.g"                                        ; Check if global.sb_leds has changed since last run/loop (comment out tho "turn off" neopixel controll)

  ; Print acceleration & input shaping control
  if global.accel_control
    
    ; Low accel on the whole first layer so that it gets good adhesion
    if global.layer_number = 1
      M204 P500                                                                ; Set printing acceleration(mm/s²)
    
    ; Everything else
    else
      if global.line_type = "External perimeter"
        if move.printingAcceleration != global.low_accel
          M204 P{global.low_accel}                                             ; Set printing acceleration(mm/s²)
        ;if move.shaping.type != global.input_shaper
          ;M593 P{global.input_shaper}                                          ; Configure input shaping
      
      elif global.line_type = "Perimeter"
        if move.printingAcceleration != 2000
          M204 P2000                                                           ; Set printing acceleration(mm/s²)
        ;if move.shaping.type != global.input_shaper
          ;M593 P{global.input_shaper}                                          ; Configure input shaping
      
      elif global.line_type = "Solid infill"
        if move.printingAcceleration != 2000
          M204 P2000                                                           ; Set printing acceleration(mm/s²)
        ;if move.shaping.type != "none"
          ;M593 P"none"                                                         ; Configure input shaping
      
      elif global.line_type = "Top solid infill"
        if move.printingAcceleration != global.low_accel
          M204 P{global.low_accel}                                             ; Set printing acceleration(mm/s²)
      
      ; When none of the above line extrusion types
      else
        if move.printingAcceleration != global.def_print_accel
          M204 P{global.def_print_accel}                                       ; Set printing acceleration(mm/s²)
        ;if move.shaping.type != "none"
          ;M593 P"none"                                                         ; Disable input shaping
  ; -----------------
  ; Daemon loop delay
  G4 S1                                                                        ; Delay running again or next command for at least 1 second

And for sake of good measure here is sb_leds-state.g as well:

; /sys/lib/led/sb_leds-state.g  v1.0
; Created by sb_leds.g to store the current/active LED colors 
; Called by daemon.g to check if global.sb_leds status has changed since last run

var sb_leds = "boot"

if var.sb_leds = global.sb_leds
  ; Same status, do nothing
else
  ; New status, change colors
  M98 P"/sys/lib/led/sb_leds.g"

Exerqtor

@dc42
Just checked in on the debug log, and it's sure been a few resets today (while i'm at work and the house is empty):

2023-06-06 06:44:52 [debug]  0
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.3+ (2023-06-05 14:42:23)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:09 [warn] HTTP client 192.168.10.100 login succeeded (session key 0)
2023-06-06 07:13:20 [warn] Date and time set at power up + 00:00:09
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.3+ (2023-06-05 14:42:23)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:08 [warn] HTTP client 192.168.10.100 login succeeded (session key 0)
2023-06-06 09:20:26 [warn] Date and time set at power up + 00:00:08
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.3+ (2023-06-05 14:42:23)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:08 [warn] HTTP client 192.168.10.100 login succeeded (session key 0)
2023-06-06 10:58:17 [warn] Date and time set at power up + 00:00:08
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.3+ (2023-06-05 14:42:23)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:07 [warn] HTTP client 192.168.10.100 login succeeded (session key 0)
2023-06-06 11:03:51 [warn] Date and time set at power up + 00:00:07
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-beta.3+ (2023-06-05 14:42:23)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
power up + 00:00:08 [warn] HTTP client 192.168.10.100 login succeeded (session key 0)
2023-06-06 13:11:17 [warn] Date and time set at power up + 00:00:08

The 06:44:52 entry is the last line of the M122 i posted early today btw.

dc42

@Exerqtor thanks. Please post your complete config,g file and any macro files that it or daemon.g call. I'll run a similar system and see if I can reproduce the problem.

Exerqtor

@dc42
Yeah OFC! I'm pretty sure this should be everything.
(Change filetype to ".rar")

I posted sb_leds-state.g inside of the last reply if that's of any interest, as you see it don't do much unless global.sb_leds changes value (which it don't when idle) lol.

EDIT:

Just as a FYI, the machine haven't reset since 13:11:17 yesterday.

Should i keep posting M122's when it eventualy resets or do you have as much info as you can for the time being?

dc42

@Exerqtor thanks, I've been running your configuration for nearly 3 hours now - hasn't reset yet.

I included a PanelDue in the setup because I saw that you had the line to enable it in config.g .

I also saw that you have enabled FTP. The comment says it is for remote backups. Do you have anything running on your PC that makes FTP requests when the printer is idle?

Exerqtor

@dc42
I haven't had any resets myself today either so I have no clue whats going on. And I have NOT changed anything just to be clear

Yeah I have a 5" PD v3 connected.

I only use ftp manually with WinSCP for the times I need to extract something, it's not automated process for backups anywhere

EDIT:

Still no more resets as of 6:40 in the morning of June 8th on my end 🤷‍♂️