3.5.0-B4+ 1LC Stack Overflow causing reset

engikeneer

Re: [3.5.0-B4](having random CAN error timeouts) https://forum.duet3d.com/topic/32694/3-5-0-b4-having-random-can-error-timeouts

@dc42 Since upgrading to the 3.5.0B4+ binary you shared and replacing my CAN cables, things seems to have been going well until today... I am trying a multi-tool print (I've only been using T0 for a while), and after a few seconds of printing with T1 doing the skirt, I hear a click from the extruder and get a CAN disconnected error from the toolboard. It is very repeatable with the short print file I am trying (two small squares next to each other). I have tried removing input shaping, but get the same issue

T0 has been printing fine and does it's first bit of the print no problem. It's just T1 that has this issue. Both tools are pretty much identical in terms of setup, wiring etc.

The toolboard seems to regain CAN sync quite quickly, but it is a few minutes before I can get a M122 B21 to respond (even when the console says it has reconnected). I got the below back after a bit. Seems like it was a software reset due to a stack overflow.

Is there anything obvious I am doing wrong or can provide to help diagnose? Config & print file below

M122 B21
Diagnostics for board 21:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4+ (2023-06-30 15:40:06)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 2396, free system stack 88 words
Tasks: Move(3,nWait,0.0%,79) HEAT(2,nWait,0.1%,81) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.0%,75) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,3.0%,57) MAIN(1,running,92.0%,412) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.8%,114), total 100.0%
Last reset 00:04:28 ago, cause: software
Last software reset at 2023-07-23 11:04, reason: StackOverflow, available RAM 3576, slot 2
Software reset code 0x0100 ICSR 0x0000000e SP 0x20007f44 Task  Freestk 3644 bad marker
Stack: 200045f8 2000462c 0001c55b 00000000 00000000 008dde8b 0001b701 20003bc4 fffffffd a5a5a5a5 00000000 00000002 00000000 00000002 0001c1af 00000000 200019b4 20001958 20001a80 000226fc 20001958 200019b4 00000032 20001ad4 00005f95 00000001 20001b30
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, read errors 0, write errors 1, ifcnt 36, reads 3160, writes 9, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 6757
Moves scheduled 83, completed 83, in progress 0, hiccups 2, segs 85, step errors 0, maxPrep 356, maxOverdue 186681744, maxInc 186681744, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.95
Peak sync jitter -4/8, peak Rx sync delay 213, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.5, current 24.5, max 24.5
MCU temperature: min 36.6C, current 36.7C, max 37.8C
Last sensors broadcast 0x00000000 found 0 25 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 2175, send timeouts 0, received 1697, lost 1939, free buffers 18, min 0, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 47715, adv -178530709/186756400
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 6, other errors 0

config

; Configuration file for Duet 3 Mini 5+ (firmware version 3.3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.3.16 on Sun Apr 02 2023 08:54:48 GMT+0100 (British Summer Time)

; General preferences
M575 P1 S1 B57600                            ; enable support for PanelDue
G90                                          ; send absolute coordinates...
M83                                          ; ...but relative extruder moves
M550 P"Loki"                                 ; set printer name
M669 K1                                      ; select CoreXY mode

; Network
M552 S1                                      ; enable network
M586 P0 S1                                   ; enable HTTP
M586 P1 S0                                   ; disable FTP
M586 P2 S0                                   ; disable Telnet

G4 S4

; Drives
M569 P0.0 S0                                 ; physical drive 0.0 goes backwards (x)
M569 P0.1 S0                                 ; physical drive 0.1 goes backwards (y)
M569 P0.2 S1                                 ; physical drive 0.2 goes forwards (z)
M569 P0.3 S1 D3 V10                                ; physical drive 0.3 goes forwards (pebble wiper)
M569 P0.4 S0                                 ; physical drive 0.4 goes backwards (coupler)
;M569 P20.0 S0                                 ; physical drive 20.0 goes backwards (T0 Motor)
M569 P21.0 S0                                 ; physical drive 20.0 goes backwards (T1 Motor)
M569 P22.0 S0                                 ; physical drive 20.0 goes backwards (T2 Motor)
;M569 P23.0 S0                                 ; physical drive 20.0 goes backwards (T3 Motor)
M584 X0.0 Y0.1 Z0.2 A0.3 C0.4 E22.0:21.0;:20.0:23.0               ; set drive mapping
M350 C8 I0									 ; configure microstepping for coupler
M350 X16 Y16 Z16 A16 E16:16 I1;:16:16 I1                      ; configure microstepping with interpolation
M92 X100 Y100 Z1600 A400 C100 E405:410;:410:410                   ; set steps per mm
M566 X500 Y500 Z100 A50 C2 E1000:1000;:1000:1000           ; set maximum instantaneous speed changes (mm/min)
M203 X20000 Y20000 Z1200 A1500 C5000 E5000:5000;:5000:5000  ; set maximum speeds (mm/min)
M201 X8000 Y8000 Z400 A400 C400 E3000:3000;:3000:3000                 ; set accelerations (mm/s^2)
M906 X1850 Y1850 Z1330 I60                      ; set motor currents (mA) and motor idle factor in percent
M906 A600 C500 E1000:1000  I20 ;:1000:1000                     ; set motor currents (mA) and motor idle factor in percent
M84 S120                                      ; Set idle timeout

; Axis Limits
M98 P"/macros/Set Machine Limits"
;M208 X0 Y0 Z0 S1                             ; set axis minima
;M208 X300 Y210 Z300 S0                       ; set axis maxima

; Endstops
M574 X1 S3                                   ; configure sensorless endstop for low end on X
M574 Y1 S3                                   ; configure sensorless endstop for low end on Y
M574 C1 S3                                   ; configure sensorless endstop for low end on C
;M574 A1 S3                                   ; configure sensorless endstop for low end on a
M574 A2 S1 P"io1.in"						; configure endstop at low end on A

; Stall Detection
M915 C S5 F0 H200 
M915 X Y S5 F0 H200
;M915 A S5 F0 H108
M569 P0.3 D3 V10 							; Set Axis A to use stealthchop

;Input Shaping
;M593 P"VZD" F44; F42.5
;M593 P"EI3" F44; F42.5
;M593 P"EI2" F44; 

; Z-Probe
M558 P5 C"io5.in" H3 F1000:150 A5 S0.02 T20000     	; set Z probe type to switch and the dive height + speeds
G31 P200 X0 Y0 Z0                            		; set Z probe trigger value, offset and trigger height
M557 X10:290 Y10:190 P20                     		; define mesh grid

; Bed Heater
M308 S0 P"temp0" Y"thermistor" A"Bed" T100000 B4725 C7.06e-8 ; configure sensor 0 as thermistor on pin temp0
M950 H0 C"out0" T0                           ; create bed heater output on out0 and map it to sensor 0
M307 H0 B0 S1.00                             ; disable bang-bang mode for the bed heater and set PWM limit
M140 H0                                      ; map heated bed to heater 0
M143 H0 S150                                 ; set temperature limit for heater 0 to 120C

; Pebble Wiper
M950 S0 C"io3.out" Q300			; Define Pebble Wiper Servo

; Tool 0
M308 S1 P"22.temp0" Y"thermistor" A"T0" T100000 B4725 C7.06e-8 ; Set thermistor
M950 H1 C"22.out0" T1                                         ; Extruder 0 heater
M143 H1 S300                                                 ; Set temperature limit for heater 1 to 300C
M950 F1 C"22.out2"
M950 F2 C"22.out1"
M106 P1 S255 H1 T70             ; T0 HE
M106 P2 S0 B0.25                     ; T0 PCF
M563 P0 S"T0 0.4mm" D0 H1 F2         ; Define tool 0
G10 P0 X0 Y0 Z0                         ; Reset tool 0 axis offsets
G10 P0 R0 S0                            ; Reset initial tool 0 active and standby temperatures to 0C

; Tool 1
M308 S2 P"21.temp0" Y"thermistor" A"T1" T100000 B4725 C7.06e-8 ; Set thermistor
M950 H2 C"21.out0" T2                                         ; Extruder 1 heater
M143 H2 S300    ; Set temperature limit for heater 2 to 300C    
M950 F3 C"21.out2"
M950 F4 C"21.out1"
M106 P3 S255 H2 T70             ; T1 HE fan thermostatic to heater 2
M106 P4 S0 B0.25                      ; T1 PCF
M563 P1 S"T1 0.4mm" D1 H2 F4         ; Define tool 1
G10 P1 X0 Y0 Z0                         ; Reset tool 1 axis offsets
G10 P1 R0 S0                            ; Reset initial tool 1 active and standby temperatures to 0C

;tool offsets
G10 P0 X23 Y41 Z-5.1				; T0 
G10 P1 X22.5 Y41 Z-5.7			; T1
;G10 P2 X22.575 Y40.98 Z-6.4		; T2 Hemera
;G10 P2 X22.15 Y40.85 Z-15.25			; T2 Hemera Volcano
;G10 P3 X23.1 Y41.1 Z-5.34			; T3 Hemera

; Pressure Advance
M572 D0 S0.03 						; pressure advance T0
M572 D1 S0.03						; pressure advance T1
;M572 D2 S0.03 						; pressure advance T2
;M572 D3 S0.03						; pressure advance T3

; Firmware Retraction
M207 P0 S0.4 F3000
M207 P1 S0.4 F3000
;M207 P2 S0.4 F3000 
;M207 P3 S0.4 F3000

; Pid Tuning
M307 H0 R1.607 K0.253:0.000 D3.06 E1.35 S1.00 B0
M307 H1 R2.556 K0.326:0.519 D6.03 E1.35 S1.00 B0 V24.1
M307 H2 R1.813 K0.362:0.007 D6.08 E1.35 S1.00 B0 V24.4

;Extra Sensors
M308 S5 Y"mcu-temp" A"MCU"  

; Fans & LED's
M950 F0 C"out_6"				; Define enclosure LED pin
M950 F9 C"out_5" Q12000				; Define board cooling fan
M106 P9 L50 X200 H5 T30:80				; Set board cooling fans to come on when MCU temp is above 40C

gcode file to recreate
Shape-Box_TC.gcode

gloomyandy

@engikeneer Could you post the output from M122 on your other toolboard. What order are the tools used in (so does the print start with tool 0 or 1)? If you swap them over do you get an error on tool 0 now or does it stay with tool 1?

engikeneer

@gloomyandy Thanks for the quick reply!

I tried running the same gcode file but with the T0 and T1 commands swapped (I used find & replace so no re-slicing). This gave the same issue with T1 losing connection part way through the skirt, but I let the print continue and T0 printed the second object fine.

Full set of M122's before and after below which I just recreated.
I should clarify that I don't get an error message in console that B21 is disconnected, just one saying that it has reconnected. I don't have any events set up, so I think this is expected.

M122 Before:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: QUS4M-U096U-D65J0-40KMD-3W03Z-ZK6H6
Used output buffers: 29 of 40 (36 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 123548 of which 12 recycled
Never used RAM 15500, free system stack 192 words
Tasks: NETWORK(2,nWait,12.5%,215) HEAT(3,nWait,0.0%,358) Move(4,nWait,0.0%,358) CanReceiv(6,nWait,0.1%,662) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,0.7%,114) MAIN(1,running,85.4%,704) IDLE(0,ready,0.5%,29) AIN(4,delaying,0.8%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:00:51 ago, cause: power up
Last software reset at 2023-07-23 11:44, reason: User, Gcodes spinning, available RAM 11980, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00487000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 51984, completed 51984, timed out 0, errs 0
MCU temperature: min 22.9, current 27.7, max 27.9
Supply voltage: min 24.4, current 24.6, max 24.6, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 2700, writes 14, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 2700, writes 14, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 2702, writes 12, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 15, reads 2698, writes 15, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 2700, writes 14, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-07-23 12:52:51
Cache data hit count 94435396
Slowest loop: 5.68ms; fastest: 0.16ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 3.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 482, received 2002, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 6, free buffers 18 (min 17), ts 260/259/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 10.32ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(2) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address f0:08:d1:02:ed:e4
Module reset reason: Power up, Vcc 3.36, flash size 2097152, free heap 39472
WiFi IP address 192.168.1.168
Signal strength -56dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 5 0 0 0 0 0 0 0

M122 B22 (T0) before

M122 B22
Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4+ (2023-06-30 15:40:06)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 3984, free system stack 142 words
Tasks: Move(3,nWait,0.0%,135) HEAT(2,nWait,0.2%,79) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.0%,77) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,nWait,3.0%,67) MAIN(1,running,91.9%,324) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.8%,114), total 100.0%
Last reset 00:00:57 ago, cause: power up
Last software reset at 2023-07-21 22:45, reason: StackOverflow, available RAM 3864, slot 2
Software reset code 0x0100 ICSR 0x0000000e SP 0x20007f44 Task HEAT Freestk 3644 bad marker
Stack: 200045f8 2000462c 0001c55b 00000000 43520000 000000ab 0001b701 20003bc4 fffffffd f6bd5ddf 00000000 00000002 00000000 00000002 0001c1af 00000000 200019b4 20001958 20001a80 000226ec 20001958 200019b4 00000032 20001ad4 00005f95 00000001 20001b30
Driver 0: pos 0, 405.0 steps/mm, standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 28571, writes 12, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -5/6, peak Rx sync delay 203, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.5
MCU temperature: min 27.0C, current 31.5C, max 31.5C
Last sensors broadcast 0x00000002 found 1 204 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 1133, send timeouts 0, received 731, lost 0, free buffers 18, min 18, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 6, other errors 0

M122 B21 (T1) before:

M122 B21
Diagnostics for board 21:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4+ (2023-06-30 15:40:06)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 3984, free system stack 142 words
Tasks: Move(3,nWait,0.0%,135) HEAT(2,nWait,0.2%,80) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.0%,77) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,3.0%,67) MAIN(1,running,91.9%,324) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.8%,114), total 100.0%
Last reset 00:00:59 ago, cause: power up
Last software reset at 2023-07-23 11:48, reason: StackOverflow, available RAM 3768, slot 2
Software reset code 0x0100 ICSR 0x0000000e SP 0x20007f44 Task HEAT Freestk 3644 bad marker
Stack: 200045f8 2000462c 0001c55b 20004810 3f06e4ad 412c0001 0001b701 20003bc4 fffffffd a5a5a5a5 00000000 00000002 00000000 00000002 0001c1af 00000000 200019b4 20001958 20001a80 000226ec 20001958 200019b4 00000032 20001ad4 00005f95 00000001 20001b30
Driver 0: pos 0, 410.0 steps/mm, standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 29806, writes 12, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -4/6, peak Rx sync delay 195, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.5, current 24.6, max 24.6
MCU temperature: min 25.5C, current 30.4C, max 30.4C
Last sensors broadcast 0x00000004 found 1 174 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 1183, send timeouts 0, received 764, lost 0, free buffers 18, min 18, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 6, other errors 0

M122 After

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: QUS4M-U096U-D65J0-40KMD-3W03Z-ZK6H6
Used output buffers: 19 of 40 (40 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 123924 of which 12 recycled
Never used RAM 11956, free system stack 135 words
Tasks: NETWORK(2,nWait,16.0%,205) HEAT(3,nWait,0.0%,352) Move(4,nWait,0.3%,244) CanReceiv(6,nWait,0.1%,662) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,351) TMC(4,nWait,0.7%,74) MAIN(1,running,81.6%,698) IDLE(0,ready,0.4%,29) AIN(4,delaying,0.8%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:10:26 ago, cause: power up
Last software reset at 2023-07-23 11:44, reason: User, Gcodes spinning, available RAM 11980, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00487000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 627056, completed 627056, timed out 0, errs 0
MCU temperature: min 27.7, current 32.2, max 35.5
Supply voltage: min 24.4, current 24.6, max 24.6, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/48/48, gc cycles 0
Events: 1 queued, 1 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 29, reads 30247, writes 15, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 29, reads 30246, writes 15, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 15, reads 30258, writes 3, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 18, reads 30259, writes 3, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 19, reads 30257, writes 5, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-07-23 13:02:26
Cache data hit count 1060572620
Slowest loop: 1000.45ms; fastest: 0.11ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.9ms, write time 2.1ms, max retries 0
=== Move ===
DMs created 83, segments created 13, maxWait 66135ms, bed compensation in use: mesh, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 967, completed 967, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.1
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 5739, received 18656, lost 0, boc 0
Longest wait 5ms for reply type 6024, peak Tx sync delay 199, free buffers 18 (min 17), ts 2875/2875/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 214.22ms; fastest: 0.06ms
Responder states: MQTT(0) HTTP(0) HTTP(2) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address f0:08:d1:02:ed:e4
Module reset reason: Power up, Vcc 2.74, flash size 2097152, free heap 37976
WiFi IP address 192.168.1.168
Signal strength -53dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

M122 B22 (T0) After:

M122 B22
Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4+ (2023-06-30 15:40:06)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 3768, free system stack 88 words
Tasks: Move(3,nWait,0.0%,71) HEAT(2,nWait,0.5%,37) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.0%,75) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,3.0%,57) MAIN(1,running,91.6%,324) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.8%,114), total 100.0%
Last reset 00:10:20 ago, cause: power up
Last software reset at 2023-07-21 22:45, reason: StackOverflow, available RAM 3864, slot 2
Software reset code 0x0100 ICSR 0x0000000e SP 0x20007f44 Task HEAT Freestk 3644 bad marker
Stack: 200045f8 2000462c 0001c55b 00000000 43520000 000000ab 0001b701 20003bc4 fffffffd f6bd5ddf 00000000 00000002 00000000 00000002 0001c1af 00000000 200019b4 20001958 20001a80 000226ec 20001958 200019b4 00000032 20001ad4 00005f95 00000001 20001b30
Driver 0: pos 0, 405.0 steps/mm, standstill, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 19625, writes 2, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 33039
Moves scheduled 319, completed 319, in progress 0, hiccups 8, segs 9, step errors 0, maxPrep 441, maxOverdue 53, maxInc 49, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 1.00
Peak sync jitter -5/7, peak Rx sync delay 216, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.1, current 24.5, max 24.5
MCU temperature: min 27.0C, current 37.1C, max 39.9C
Last sensors broadcast 0x00000002 found 1 190 ticks ago, 0 ordering errs, loop time 1
CAN messages queued 11320, send timeouts 0, received 6193, lost 0, free buffers 18, min 18, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 347, adv 35667/74662
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 0, other errors 0

M122 B21 (T1) After

M122 B21
Diagnostics for board 21:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4+ (2023-06-30 15:40:06)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 2396, free system stack 88 words
Tasks: Move(3,nWait,0.0%,79) HEAT(2,nWait,0.1%,81) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.0%,75) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,3.0%,57) MAIN(1,running,92.0%,410) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.8%,114), total 100.0%
Last reset 00:06:05 ago, cause: software
Last software reset at 2023-07-23 12:56, reason: StackOverflow, available RAM 3768, slot 0
Software reset code 0x0100 ICSR 0x0000000e SP 0x20007f44 Task HEAT Freestk 3644 bad marker
Stack: 200045f8 2000462c 0001c55b 00000000 00000002 00000066 0001b701 20003bc4 fffffffd f4a9595a 00000000 00000002 00000000 00000002 0001c1af 00000000 200019b4 20001958 20001a80 000226ec 20001958 200019b4 00000032 20001ad4 00005f95 00000001 20001b30
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, read errors 0, write errors 1, ifcnt 24, reads 51893, writes 10, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 6723
Moves scheduled 83, completed 83, in progress 0, hiccups 2, segs 85, step errors 0, maxPrep 356, maxOverdue 188842993, maxInc 188842993, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.97
Peak sync jitter -4/7, peak Rx sync delay 209, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.5, current 24.6, max 24.6
MCU temperature: min 33.1C, current 33.3C, max 38.1C
Last sensors broadcast 0x00000000 found 0 242 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 2954, send timeouts 0, received 2937, lost 1983, free buffers 18, min 0, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 47454, adv -180786252/188917586
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 6, other errors 0

gloomyandy

@engikeneer said in 3.5.0-B4+ 1LC Stack Overflow causing reset:

Thanks for that, is there anything different about your two tools? Extra fans, heaters? One has Neopixels or anything like that?

I can't see anything obvious when looking at the config.g and the m122 outputs, but the "after" case for T1 does show a smaller amount of free memory and also the following:

CAN messages queued 2954, send timeouts 0, received 2937, lost 1983, free buffers 18, min 0, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 47454, adv -180786252/188917586

It is hard to be sure if this is part of the problem or simply the result of the board crashing and restarting. Perhaps @dc42 will have some thoughts?

engikeneer

@gloomyandy no T0 and T1 are the same (it's an e3D TC with hemeras, 0.4mm v6 etc). Only difference I can think is that T0 has a hotend grounding resistor wired in, which T1 doesn't, but that has not been an issue in the past...

dc42

@engikeneer I think what may be happening is that a debug message is being generated, and owing to a change in the way that debug messages are processed the Heat task stack is now too small when this happens. Please try the new tool board firmware at https://www.dropbox.com/scl/fo/sj9kuloenbp6e70asnxue/h?rlkey=upykfuquc574l61deqwo94xr3&dl=0.

engikeneer

@dc42 Thanks for providing this. I have tested and it does solve the crashing issue, and explains why there was an issue with the heat task stack... I was getting a heater fault occurring on T1. It may have also not helped that I didn't have a heater-fault.g file. Now I just need to work out why the fault was happening...

dc42

@engikeneer thanks. A debug message was being generated for some types of heater fault. In previous builds they were suppressed if the build configuration didn't specify a serial debug port, but now they are relayed over CAN to the main board instead.

The amount of free RAM will have decreased because of the increased stack size, so please monitor "Never used RAM" in the M122 report for the tool board.

engikeneer

@dc42 Good news is that I don't seem to be having any RAM issues Never used RAM seems to be steady at 2728:

M122 B22
Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4+ (2023-07-26 12:19:38)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 2728, free system stack 88 words
Tasks: Move(3,nWait,0.0%,71) HEAT(2,nWait,0.3%,87) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.1%,75) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,3.0%,57) MAIN(1,running,91.8%,324) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.8%,114), total 100.0%
Last reset 00:45:46 ago, cause: power up
Last software reset at 2023-07-21 22:45, reason: StackOverflow, available RAM 3864, slot 2
Software reset code 0x0100 ICSR 0x0000000e SP 0x20007f44 Task HEAT Freestk 3644 bad marker
Stack: 200045f8 2000462c 0001c55b 00000000 43520000 000000ab 0001b701 20003bc4 fffffffd f6bd5ddf 00000000 00000002 00000000 00000002 0001c1af 00000000 200019b4 20001958 20001a80 000226ec 20001958 200019b4 00000032 20001ad4 00005f95 00000001 20001b30
Driver 0: pos 0, 410.0 steps/mm, ok, SG min 0, read errors 0, write errors 0, ifcnt 25, reads 61617, writes 25, timeouts 4, DMA errors 0, CC errors 0, failedOp 0x6a, steps req 0 done 205224
Moves scheduled 2427, completed 2427, in progress 0, hiccups 45, segs 41, step errors 0, maxPrep 1361, maxOverdue 60251, maxInc 13299, mcErrs 0, gcmErrs 0, ebfmin -0.72 max 1.00
Peak sync jitter -5/8, peak Rx sync delay 300, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.3, current 24.3, max 24.5
MCU temperature: min 23.2C, current 46.6C, max 46.6C
Last sensors broadcast 0x00000008 found 1 241 ticks ago, 0 ordering errs, loop time 1
CAN messages queued 55026, send timeouts 0, received 60157, lost 0, free buffers 18, min 17, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 483, adv 35679/74655
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 6, other errors 0

One thing I have noticed with the 3.5.0-B4+ build you provided is that I get a false error appearing when homing my printer during a print. I think this was present in the previous B4+ build you provided but can't be sure.

Error: Failed to home axes

I created a print file with only G28 in it, and I get that error (despite all axes homing fine). If I run a G28 normally, then I get no error.
This isn't causing any particular issue, it's just annoying getting the false error messages. My homing files below - happy to provide the sub-macros if needed
Thanks

homeall

; homeall.g
; called to home all axes
;
; generated by RepRapFirmware Configuration Tool v3.2.3 on Sat May 29 2021 21:13:21 GMT+0100 (British Summer Time)

M98 P"homec.g"			; Home C (ToolHead)

M98 P"homea.g"			; Home A (Pebble Wiper)

M98 P"homey.g"			; Home Y

M98 P"homex.g"			; Home X

M98 P"homez.g"			; Home Z

homec

; homec.g
; called to home the C axis
;

;from old

G92 C260
M913 C40			; C MOTOR TO 40% CURRENT
G1 C-260 F2400  ; drive the C-axis to the stop
M913 C100			; C MOTOR TO 100% CURRENT
G1 C1 F50000
G92 C0

;Open Coupler
M98 P"/macros/Tool - Unlock"

homea

;called to home axis A

G91               ; relative positioning
G1 H1 A50 F1500 ; move quickly to A axis endstop and stop there (first pass)
G1 A-2 F2000       ; go back a few mm
G1 H1 A50 F360  ; move slowly to A axis endstop once more (second pass)
G90               ; absolute positioning

homex

; homex.g
; called to home the X axis
;
G91 			; use relative positioning
M98 P"/macros/Set Machine Limits"

M400 			; make sure everything has stopped before we make changes
M569 P0.0 D3 V10 
M569 P0.1 D3 V10 
G4 P100			; wait 100ms
M17 X Y						    ; Energize Motors X Y
M913 X50 Y50 		; drop motor currents to 50%
M915 H200 X Y S0 R0 F0 	; set X and Y to sensitivity 0, do nothing when stall, unfiltered
G4 P100
G1 H2 Z3 F5000		; lift Z 3mm
G1 H2 X1 Y1 F3000		; energise to avoid false triggers during sensorless homing
M400
G4 P100	
G1 H1 X-400 F6000 	; move left 400mm, stopping at the endstop
G1 H1 X2 F2000 		; move away from endstop
G1 H2 Z-3 F1200		; lower Z

G90 			; back to absolute positioning
M400 			; make sure everything has stopped before we reset the motor currents
M913 X100 Y100 		; motor currents back to 100%
M569 P0.0 D2
M569 P0.1 D2

homey

; homey.g
; called to home the Y axis
;

G91 			; use relative positioning
M98 P"/macros/Set Machine Limits"
M400 			; make sure everything has stopped before we make changes
M569 P0.0 D3 V10 
M569 P0.1 D3 V10 
G4 P100			; wait 100ms
M17 X Y						    ; Energize Motors X Y
M913 X50 Y50 		; drop motor currents to 50%
M915 H200 X Y S0 R0 F0 	; set X and Y to sensitivity 0, do nothing when stall, unfiltered
G4 P100
G1 H2 Z3 F5000		; lift Z 3mm
G1 H2 X1 Y1 F3000		; energise to avoid false triggers during sensorless homing
M400
G4 P100		
G1 H1 Y-400 F6000 	; move left 400mm, stopping at the endstop
G1 H1 Y2 F2000 		; move away from end
G1 H2 Z-3 F1200		; lower Z

G90 			; back to absolute positioning
M400 			; make sure everything has stopped before we reset the motor currents
M913 X100 Y100 		; motor currents back to 100%
M569 P0.0 D2
M569 P0.1 D2

homez

; homez.g
; called to home the Z axis

T-1       				;just in case there is a tool coupled, go try to drop it at the dock
M116 H0 S1				; Wait for bed to reach temp;
G29 S2 					; Disable mesh compensation
G91 					; Relative mode
G1 H2 Z5 F5000			; Lower the bed
G90						; back to absolute positioning
G1 X150 Y100 F50000		; Position the endstop above the bed centre
G91 					; Relative mode
G4 P1000				; wait 1000msec
G30 					; probe
G1 Z20 F5000			; Drop the Bed
G90						; Back to absolute positioning

gloomyandy

@engikeneer I think it would help if you try manually homing each axis in turn to identify which one is producing that error message.

engikeneer

@gloomyandy good thinking - I can confirm it is the Z homing that causes the issue. I homed all axes then one-by-one ran print jobs with just G28 X, G28 Y etc in and it was G28 Z that triggered it. I also ran it with commenting out a few lines in my homez and I think the one causing the issue is the M116:

; homez.g
; called to home the Z axis

T-1       			;just in case there is a tool coupled, go try to drop it at the dock
M116 H0 S1			; Wait for bed to reach temp;  <<<< THIS ONE
G29 S2 				; Disable mesh compensation
G91 				; Relative mode
G1 H2 Z5 F5000			; Lower the bed
G90				; back to absolute positioning
G1 X150 Y100 F50000		; Position the endstop above the bed centre
G91 				; Relative mode
G4 P1000			; wait 1000msec
G30 				; probe
G1 Z20 F5000			; Drop the Bed
G90				; Back to absolute positioning

dc42

@engikeneer when you get that error message, is the bed heating up?

Do you also get that error message if you command the bed to heat manually and then command Z homing manually?

engikeneer

@dc42 I get the error message whether the bed is off, heating up, or already at temp. I also changed it to a simple M116 (i.e. no H0 S1) and same result.

If I do any of this not in a print (i.e. just manually by hitting homez in DWC, or putting G28 Z in the console), I do not get the error.

dc42

@engikeneer I believe this is fixed in 3.5.0-rc.1.