Tool failed during print

[oliof]

@jumpedwithbothfeet said in Tool failed during print:

Last software reset at 2024-04-21 11:59, reason: OutOfMemory, available RAM 0, slot 0

We would likely need to see your config.g and know more about the toolboard to understand more about why the toolboard ran out of memory

(also, probably worth updating to 3.5.1 while you are at it)

[dc42]

@jumpedwithbothfeet it ran out of memory while trying to allocate some for a move. Your thermistor has nothing to do with it.

Have you used M950 to allocate a Neopixel port on the tool board? If so then you may need to reduce the maximum number LEDs supported in that M950 command.

[jumpedwithbothfeet]

@dc42 @oliof Here is my config.g, I don't have anything attached to the tool board other than a part fan/heater/thermistor/end stop, I've messed around with neopixels but always off the main board.

I have just updated to 3.5.1, I`m guessing during this process a "reset" is carried out, would it be worth carrying out a factory reset?

; Configuration file for Duet 3 (firmware version 3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v2.1.8 on Mon Apr 06 2020 18:55:28 GMT+0100 (British Summer Time)
 
; General preferences
G90                                         ; send absolute coordinates...
M83                                         ; ...but relative extruder moves
M550 P"Duet 3"                              ; set printer name
G4 S2                                       ;wait for expansion boards to start
M669 K1                                     ; select CoreXY mode
M575 P1 S1 B57600                           ; paneldue
;M552 S1                                     ; Enable networking
 
; Drives
M569 P0.0 S0 ;D3 H50 V50                                      ; physical drive 0.0 goes forwards (BL-Z)
M569 P0.1 S0 ;D3 H5 V5                                        ; physical drive 0.1 goes Backwards (LT)
M569 P0.2 S0 ;D3 H50 V50                                      ; physical drive 0.2 goes forwards (BR-Z)
M569 P0.3 S1 ;D3 H50 V50                                      ; physical drive 0.3 goes backwards (FR-Z)
M569 P0.4 S0 ;D3 H5 V5                                        ; physical drive 0.4 goes Forwards (RT)
M569 P0.5 S0 ;D3 H50 V50                                      ; physical drive 0.5 goes forwards (FL-Z)
M569 P121.0 S0                                      ; physical drive tool board 1.1 goes forwards (E)
;M569 P1.0 S0                                        ; Physical drive 3HC driver 0 goes forwards (E1)
;M569 P1.1 S1                                        ; Physical drive 3HC driver 1 goes backwards (E2)
;M569 P1.2 S1                                        ; Physical drive 3HC driver 2 goes backwards (U)
M584 X0.4 Y0.1 Z0.0:0.2:0.3:0.5 E121.0     ; set drive mapping
M671 X-35:395:395:-35 Y420:420:-10:-10 S10          ; Z leadscrews locations
M350 X16 Y16 Z16 E16 I1                             ; configure microstepping with interpolation
M92 X80.00 Y80.00 Z800.00 E860                    ; set steps per mm
M566 X900.00 Y900.00 Z12.00 E120.00                 ; set maximum instantaneous speed changes (mm/min)
M203 X27720 Y27720 Z600.00 E3000.00                 ; set maximum speeds (mm/min)
M201 X10000.00 Y10000.00 Z20 E5000.00               ; set accelerations (mm/s^2)
M906 X2000 Y2000 Z950 E800 I20                      ; set motor currents (mA) and motor idle factor in per cent
M84 S30                                             ; Set idle timeout
;M915 P0:1:2:3:4:5 T50
 
; Axis Limits
M208 X0 Y0 Z0 S1                            ; set axis minima
M208 X360 Y360 Z300 S0                      ; set axis maxima
 
; Endstops
M574 X2 S1 P"io4.in"                        ; configure active-high endstop for low end on X via pin io4.in
M574 Y2 S1 P"io5.in"                        ; configure active-high endstop for low end on Y via pin io5.in
M574 Z1 S2                                  ; configure Z-probe endstop for low end on Z
 
; Z-Probe
M950 S0 C"nil"                                      ; create servo pin 0 for Klicky
M558 P8 C"121.io2.in" H3 F80:50 T6000 R1 A3 S0.005  ; set Z probe type to Klicky and the dive height + speeds
G31 P50 X0 Y-30 Z13.426                              ; set Z probe trigger value, offset and trigger height
M557 X50:320 Y40:300 S45:52                         ; define mesh grid
;13.341
; Heaters
M308 S0 P"temp0" Y"thermistor" A"Bed" T98801 B4185                      ; configure sensor 0 as thermistor on pin temp0
M950 H0 C"out0" T0                                                      ; create bed heater output on out0 and map it to sensor 0
M143 H0 S130                                                            ; set temperature limit for heater 0 to 120C
M307 H0 B0 S1.00                                                        ; disable bang-bang mode for the bed heater and set PWM limit
M140 H0                                                                 ; map heated bed to heater 0
;M308 S1 P"121.TEMP_0" Y"pt1000" A"Hotend"; sensor 1 (toolboard)
M308 S1 P"121.TEMP_0" Y"thermistor" A"Hotend" T100000 B4725 C7.06e-8    ; configure sensor 1 as thermistor on pin temp2
M950 H1 C"121.out0" T1                                                  ; create nozzle heater output on out1 and map it to sensor 1
M143 H1 S300                                                            ; set temperature limit for heater 1 to 350C
M307 H1 B0 S1.00                                                        ; disable bang-bang mode for heater  and set PWM limit
M308 S2 P"temp1" Y"thermistor" A"Enclosure temp" T100000 B4725 C7.06e-8 ; configure sensor 3 as thermistor on pin temp1
M308 S3 P"mcu-temp" Y"mcu-temp" A"mcu-temp"                             ; configure sensor 2 as thermistor on pin temp2
;M308 S1 P"spi.cs0" Y"rtd-max31865"                                     ; define temperature sensor number 1 as a PT100 on the first port of a temperature daughter
 
; Fans
M950 F1 C"out4+out4.tach"                                 ; create fan 1 on pin out.1 and set its frequency (Hotend coolant fan)
M106 P1 H1 S0 L1 T50                                      ; set fan 1 value. Thermostatic control is turned on
M950 F0 C"121.out1" Q255                                  ; create fan 0 on pin out1(toolboard) and set its frequency (New parts fan)
M106 P0 S0 H-1 C"Part cooling fan"                        ; set fan 0 value. Thermostatic control is turned off
M950 F2 C"out9" Q500                                      ; create fan 2 on pin out.2  and set its frequency (CPU fan)
M106 P2 H3 T40 C"CPU fan"                                 ; set fan 2 value. Thermostatic control is turned on
 
; Lighting
M950 P3 C"io1.out" Q500                                  ; create fan 0 on pin io1.out and set its frequency (LED1)
M42 P3 S0 C"LED"                                         ; set LED 0 value. Thermostatic control is turned off
; Neopixels
M950 E0 C"led" T1 Q3000000   ; create a RGB Neopixel LED strip on the LED port and set SPI frequency to 3MHz 
 
; Tools
M563 P0 S"Water dragon" D0 H1 F0                      ; define tool 0
;M563 P1 S"V6" D0:2 H1 F0                      ; define tool 1
M568 P0 X0 Y0 Z0                             ; set tool 0 axis offsets
M568 P0 R0 S0                                ; set initial tool 0 active and standby temperatures to 0C
;G10 P1 X0 Y0 Z0                             ; set tool 1 axis offsets
;G10 P1 R0 S0                                ; set initial tool 1 active and standby temperatures to 0C
;M567 P0 E1.00:1.00
;M567 P1 E1.00:1.00
 
; Custom settings are not defined
M912 P0 S-7.9
M955 P121.0 I05
M593 P"zvdd" F61.5  ; use ZVDD input shaping to cancel ringing at 61.5Hz
 
; Miscellaneous
M501                                        ; load saved parameters from non-volatile memory
T0                                          ; select first tool
 

[oliof]

@jumpedwithbothfeet no need to do a factory reset, the configuration is applied from the main board after bootup. So just reboot. Just make sure the board really is up to date running M115 B121 after running the update procedure.

You can try to see if you can reproduce the issue running a simulation of a print after updating everything to 3.5.1.

[jumpedwithbothfeet]

@oliof thanks, I have updated to 3.5.1 and confirmed the board is also up to date, I even updated the boot loader from 2.3 to 2.8, also cleaned out any Neo pixel reference from my config and am now running a print (2hr) and all is well so far, but again this has happened before and I thought it was the thermistor playing up.

@dc42 is there anything else that will cause it to use its memory up?

#I was running the failed print at 120% speed factor would that cause an issue?

[dc42]

@jumpedwithbothfeet please run M122 B121 after the print has been running for some time, or after it has finished, to see how much memory it had left after printing and whether there was anything else unusual..

[jumpedwithbothfeet]

@dc42 Print has been running over an hour and is currently still printing without issue with about 40 minutes left I guesstimate!

M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.1 (2024-04-19 14:42:41)
Bootloader ID: SAMC21 bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 3000, free system stack 71 words
Tasks: Move(3,nWait 7,0.4%,85) HEAT(2,nWait 6,0.4%,91) CanAsync(5,nWait 4,0.0%,49) CanRecv(3,nWait 1,0.1%,71) CanClock(5,nWait 1,0.0%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,nWait 6,3.6%,53) MAIN(1,running,90.7%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.8%,112), total 100.0%
Owned mutexes:
Last reset 02:16:14 ago, cause: software
Last software reset at 2024-04-21 11:59, reason: OutOfMemory, available RAM 0, slot 0
Software reset code 0x01c0 ICSR 0x00000000 SP 0x20004c68 Task Move Freestk 115 ok
Stack: 20004ee0 000063e3 20004f40 20005fa8 20004f40 0000a51d 20004ccc 00008faf c5784ca0 2e89a8bb 2e89a8bb 47a79800 be3335d3 ce2ae763 41661135 00014f30 00000001 0001f46a 200026cc 20005fa8 200076b0 000492ca 20004ccc 00009629 000007d0 3e5c209d 3f4c1414
Driver 0: pos 0, 860.0 steps/mm, ok, SG min 0, read errors 1, write errors 0, ifcnt 49, reads 43947, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 3022477
Moves scheduled 94598, completed 94596, in progress 1, hiccups 240, segs 47, step errors 0, maxLate 1 maxPrep 757, maxOverdue 98, maxInc 64, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 1.00
Peak sync jitter 6/10, peak Rx sync delay 261, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.1, current 24.3, max 24.6
MCU temperature: min 37.5C, current 87.2C, max 87.4C
Last sensors broadcast 0x00000002 found 1 177 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 46577, send timeouts 0, received 54900, lost 0, errs 0, boc 0, free buffers 18, min 17, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 399, adv 36241/74663
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 0, contentions 0, other errors 0

is that MCU temp alittle to much?!!

[jumpedwithbothfeet]

@dc42 Print finished M122 results

M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.1 (2024-04-19 14:42:41)
Bootloader ID: SAMC21 bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 3000, free system stack 71 words
Tasks: Move(3,nWait 7,0.7%,85) HEAT(2,nWait 6,0.4%,91) CanAsync(5,nWait 4,0.0%,49) CanRecv(3,nWait 1,0.2%,71) CanClock(5,nWait 1,0.0%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,nWait 6,3.6%,53) MAIN(1,running,90.3%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.9%,112), total 100.0%
Owned mutexes:
Last reset 02:47:04 ago, cause: software
Last software reset at 2024-04-21 11:59, reason: OutOfMemory, available RAM 0, slot 0
Software reset code 0x01c0 ICSR 0x00000000 SP 0x20004c68 Task Move Freestk 115 ok
Stack: 20004ee0 000063e3 20004f40 20005fa8 20004f40 0000a51d 20004ccc 00008faf c5784ca0 2e89a8bb 2e89a8bb 47a79800 be3335d3 ce2ae763 41661135 00014f30 00000001 0001f46a 200026cc 20005fa8 200076b0 000492ca 20004ccc 00009629 000007d0 3e5c209d 3f4c1414
Driver 0: pos 0, 860.0 steps/mm, standstill, SG min 0, read errors 1, write errors 0, ifcnt 49, reads 39052, writes 0, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x72, steps req 0 done 1524175
Moves scheduled 140397, completed 140397, in progress 0, hiccups 255, segs 47, step errors 0, maxLate 0 maxPrep 706, maxOverdue 79, maxInc 64, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 1.00
Peak sync jitter 6/10, peak Rx sync delay 228, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.5, max 24.6
MCU temperature: min 37.5C, current 86.9C, max 87.4C
Last sensors broadcast 0x00000002 found 1 171 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 17411, send timeouts 0, received 35298, lost 0, errs 0, boc 0, free buffers 18, min 18, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 414, adv 36289/74656
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 0, contentions 0, other errors 0

[Phaedrux]

@jumpedwithbothfeet said in Tool failed during print:

MCU temperature: min 37.5C, current 87.2C, max 87.4C

That is pretty toasty. Is this an enclosed printer?

[jumpedwithbothfeet]

@Phaedrux yes it is but only passively heated, water cooled hot end, Im trying to figure out a way of water cooling the board as a fan won’t cut it! And I’m guessing the board rear side is where the heat gets transferred like the main board?

[jumpedwithbothfeet]

Update, found a crimp that had gone bad on the part cooling fan (ground side) this maybe partly to blame, but doesn’t solve the hot cpu issue, could it cause the memory issue?

[Phaedrux]

does it still get hot like that without the bed heater and extruder heater going and the enclosure open?

[jumpedwithbothfeet]

@Phaedrux so essentially the printer turned on but not running a print? I’ve not tried it, I’ll turn it on and report back

[jumpedwithbothfeet]

@Phaedrux the temperature increased from 23.3 degrees to 31.4 over an hour.

 
M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.1 (2024-04-19 14:42:41)
Bootloader ID: SAMC21 bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 4128, free system stack 132 words
Tasks: Move(3,nWait 7,0.0%,135) HEAT(2,nWait 6,0.3%,107) CanAsync(5,nWait 4,0.0%,55) CanRecv(3,nWait 1,0.0%,71) CanClock(5,nWait 1,0.0%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,delaying,3.4%,53) MAIN(1,running,91.6%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.6%,112), total 100.0%
Owned mutexes:
Last reset 01:07:56 ago, cause: power up
Last software reset at 2024-04-21 11:59, reason: OutOfMemory, available RAM 0, slot 0
Software reset code 0x01c0 ICSR 0x00000000 SP 0x20004c68 Task Move Freestk 115 ok
Stack: 20004ee0 000063e3 20004f40 20005fa8 20004f40 0000a51d 20004ccc 00008faf c5784ca0 2e89a8bb 2e89a8bb 47a79800 be3335d3 ce2ae763 41661135 00014f30 00000001 0001f46a 200026cc 20005fa8 200076b0 000492ca 20004ccc 00009629 000007d0 3e5c209d 3f4c1414
Driver 0: pos 0, 860.0 steps/mm, standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 44134, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter 5/8, peak Rx sync delay 204, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.4
MCU temperature: min 15.5C, current 31.2C, max 31.4C
Last sensors broadcast 0x00000002 found 1 23 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 1799, send timeouts 0, received 802, lost 0, errs 0, boc 0, free buffers 18, min 18, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 0, contentions 0, other errors 0
 

[Phaedrux]

I guess that eliminates a failed component getting hot and cooking the rest of it. The driver chip and it's supporting electronics would be the main source of heat on the board.

Can you try running a print without the enclosure sealed to see if it stays cooler/lasts longer?

[jumpedwithbothfeet]

@Phaedrux I’ve just run a 1hr print door wide open the MCU crept up consistently during the print it was 25 degrees at the start, 68.5 just before finishing so it’s getting warm, I’ll sort out a 2 hour print tomorrow to see if it continues to climb or plateaus again with the door open.

 
M122 b121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.1 (2024-04-19 14:42:41)
Bootloader ID: SAMC21 bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 3096, free system stack 71 words
Tasks: Move(3,nWait 7,0.3%,89) HEAT(2,nWait 6,0.4%,91) CanAsync(5,nWait 4,0.0%,49) CanRecv(3,nWait 1,0.1%,71) CanClock(5,nWait 1,0.0%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,nWait 6,3.5%,53) MAIN(1,running,91.1%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.7%,112), total 100.0%
Owned mutexes:
Last reset 01:22:03 ago, cause: power up
Last software reset at 2024-04-21 11:59, reason: OutOfMemory, available RAM 0, slot 0
Software reset code 0x01c0 ICSR 0x00000000 SP 0x20004c68 Task Move Freestk 115 ok
Stack: 20004ee0 000063e3 20004f40 20005fa8 20004f40 0000a51d 20004ccc 00008faf c5784ca0 2e89a8bb 2e89a8bb 47a79800 be3335d3 ce2ae763 41661135 00014f30 00000001 0001f46a 200026cc 20005fa8 200076b0 000492ca 20004ccc 00009629 000007d0 3e5c209d 3f4c1414
Driver 0: pos 0, 860.0 steps/mm, ok, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 35276, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 46438
Moves scheduled 59221, completed 59218, in progress 1, hiccups 57, segs 43, step errors 0, maxLate 0 maxPrep 504, maxOverdue 65, maxInc 64, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 1.00
Peak sync jitter 4/9, peak Rx sync delay 204, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.3, current 24.4, max 24.6
MCU temperature: min 15.8C, current 68.3C, max 68.5C
Last sensors broadcast 0x00000002 found 1 157 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 1450, send timeouts 0, received 1488, lost 0, errs 0, boc 0, free buffers 18, min 17, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 363, adv 36385/74657
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 0, contentions 0, other errors 0
I
 

[jumpedwithbothfeet]

@Phaedrux just about to finish a 3 hour print, the temp with the door open reached 73.1 so still fairly toasty! I’m gonna try it again with the cable cover off see if it makes a difference….

 
M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.1 (2024-04-19 14:42:41)
Bootloader ID: SAMC21 bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 3144, free system stack 71 words
Tasks: Move(3,nWait 7,0.3%,85) HEAT(2,nWait 6,0.4%,91) CanAsync(5,nWait 4,0.0%,49) CanRecv(3,nWait 1,0.1%,71) CanClock(5,nWait 1,0.0%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,delaying,3.5%,53) MAIN(1,running,91.0%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.7%,112), total 100.0%
Owned mutexes:
Last reset 03:45:34 ago, cause: power up
Last software reset at 2024-04-21 11:59, reason: OutOfMemory, available RAM 0, slot 0
Software reset code 0x01c0 ICSR 0x00000000 SP 0x20004c68 Task Move Freestk 115 ok
Stack: 20004ee0 000063e3 20004f40 20005fa8 20004f40 0000a51d 20004ccc 00008faf c5784ca0 2e89a8bb 2e89a8bb 47a79800 be3335d3 ce2ae763 41661135 00014f30 00000001 0001f46a 200026cc 20005fa8 200076b0 000492ca 20004ccc 00009629 000007d0 3e5c209d 3f4c1414
Driver 0: pos 0, 860.0 steps/mm, ok, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 36389, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 698704
Moves scheduled 203158, completed 203157, in progress 1, hiccups 853, segs 41, step errors 0, maxLate 0 maxPrep 567, maxOverdue 92, maxInc 64, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 1.00
Peak sync jitter 5/9, peak Rx sync delay 236, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.1, current 24.3, max 24.6
MCU temperature: min 16.1C, current 71.9C, max 73.1C
Last sensors broadcast 0x00000002 found 1 117 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 19890, send timeouts 0, received 21397, lost 0, errs 0, boc 0, free buffers 18, min 18, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 369, adv 36230/74658
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 0, contentions 0, other errors 0
 

[jumpedwithbothfeet]

No difference with the cover off!