3.5.0-B4, having random CAN error timeouts

dc42

@Nuramori the most likely reason is a bad CAN connection between the tool board and the main board.

When this happens, does the status LED on the tool board flash in sync with the one on the main board, or flash faster?
Is this issue definitely new since you upgraded to 3.5.0-beta.4 ?

Nuramori

@dc42 It's consistently happened only (so far) with this one print. I have since it occured, printed other things successfully now. It IS a long print (11 hours). Everything goes back to normal once I reset.

I'll try again and if it fails, I'll look for the lights status.

Nuramori

@dc42 it’s still failing, but I again neglected to check the lights. This time the tool head will continue to move, with the fan on, but the extruder temp reports 2000c and goes inactive, but continues to make the position moves along all axis. This appears to be happening at about the same point in this particular file. I have printed other files as a check, and they seem to work fine. The point of failure seems to be approximately where a support skin is formed and is about to make the first parts skin on top of that, meaning that separation layer where a bond is to be weak or non existent (I assume why the fan was on). Before printing I had redone an auto tune which was successful.

I did not get any can disconnect messages.

deckingman

@Nuramori For info, I've just started commissioning a new printer using a 6HC and a single tool board (latest version). I kept losing CAN communication on the tool board - verified by the fact that the sync LED would flash quickly. Pressing on the JST ZH connector would restore synchronization. I had used the supplied made up connector. I bought some new made up connectors and fitted one of these in place of the supplied one and thus far the issue seems to be resolved. I'm on an earlier firmware version so this may not be relevant to your situation.

Nuramori

@deckingman It's really strange. it's ONLY this one file that causes the CAN timeout. I've printed now, 8 files, some smaller some larger and move complex with rapid movements, and none have caused the error this one file does. I narrowed it down to where it might happen in the gcode, but nothing looks odd. (Z3.4-ish) in the file. It's frustrating, as this is the last part I need for a project.

Maybe you guys can discover something in it. - problem gcode?

Nuramori

@dc42 I was able to finally capture the data - unless I read it wrong, it's an out of memory fault? Here's the M122 I was able to get for both the main board and toolboard.

6/19/2023, 7:20:14 AM 	Diagnostics for board 20:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4 (2023-06-08 16:22:30)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 660, free system stack 142 words
Tasks: Move(3,nWait,0.0%,119) HEAT(2,nWait,0.1%,101) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.0%,75) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,3.0%,57) MAIN(1,running,92.0%,438) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.9%,142), total 100.0%
Last reset 00:00:22 ago, cause: software
Last software reset at 2023-06-19 07:19, reason: OutOfMemory, available RAM 16, slot 0
Software reset code 0x01c0 ICSR 0x00000000 SP 0x200054a8 Task Move Freestk 137 ok
Stack: 200057b0 000062df 00000000 20005114 30e4cc09 00009f99 200057c8 00007d19 0001eaea 388e7c4b 449dc61b 00000002 3db25835 360d022f 200057f8 200057e0 200068c4 00000000 464c15d3 31a06276 31a06276 2000554c 20006908 20005108 00000000 00008c91 00000000
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 2, read errors 0, write errors 1, ifcnt 29, reads 11416, writes 9, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 28, completed 0, in progress 1, hiccups 0, step errors 0, maxPrep 350, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 1, ebfmin 0.00, ebfmax 0.00
Peak sync jitter 5/9, peak Rx sync delay 207, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.3
MCU temperature: min 56.8C, current 56.8C, max 59.4C
Last sensors broadcast 0x00000000 found 0 144 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 209, send timeouts 0, received 245, lost 0, free buffers 18, min 0, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 34260, adv -1177163/957204334
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 3, other errors 0




6/19/2023, 7:20:14 AM 	=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.5.0-beta.4 (2023-06-08 23:41:30) running on Duet 3 MB6HC v1.0 or earlier (SBC mode)
Board ID: 08DJM-956L2-G43S4-6J1F2-3SJ6T-1A6QH
Used output buffers: 1 of 40 (18 max)
=== RTOS ===
Static ram: 155012
Dynamic ram: 87164 of which 4080 recycled
Never used RAM 76264, free system stack 122 words
Tasks: SBC(2,rWait:,18.0%,422) HEAT(3,nWait,0.4%,323) Move(4,nWait,26.1%,214) CanReceiv(6,nWait,0.5%,771) CanSender(5,nWait,0.4%,328) CanClock(7,delaying,0.1%,349) TMC(4,nWait,155.7%,59) MAIN(2,running,339.1%,135) IDLE(0,ready,0.6%,30), total 540.8%
Owned mutexes: Autopause(MAIN)
=== Platform ===
Last reset 06:43:25 ago, cause: power up
Last software reset at 2023-06-15 03:41, reason: User, Gcodes spinning, available RAM 75808, slot 0
Software reset code 0x6003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0044a000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
MCU temperature: min 31.4, current 39.0, max 40.4
Supply voltage: min 22.7, current 23.4, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0
Heap OK, handles allocated/used 99/7, heap memory allocated/used/recyclable 2048/276/164, gc cycles 0
Events: 1 queued, 0 completed
Driver 0: ok, SG min 0, mspos 440, reads 29458, writes 21 timeouts 0
Driver 1: ok, SG min 0, mspos 808, reads 29458, writes 21 timeouts 0
Driver 2: ok, SG min 0, mspos 522, reads 29454, writes 25 timeouts 0
Driver 3: ok, SG min 0, mspos 170, reads 29454, writes 25 timeouts 0
Driver 4: ok, SG min 0, mspos 936, reads 29458, writes 21 timeouts 0
Driver 5: standstill, SG min 0, mspos 8, reads 29468, writes 11 timeouts 0
Date/time: 2023-06-19 07:20:12
Slowest loop: 93.93ms; fastest: 0.05ms
=== Storage ===
Free file entries: 20
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 59, maxWait 2492950ms, bed compensation in use: mesh, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 322093, completed 322093, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 4], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP* is idle in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause* is doing "M122" in state(s) 0 74, running macro
File2* is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x80000007
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 522974, received 483920, lost 0, boc 0
Longest wait 3ms for reply type 6013, peak Tx sync delay 44962, free buffers 50 (min 49), ts 121028/121027/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 39804/39804
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x25ce8
Buffer RX/TX: 2696/2728-0, open files: 0
=== Duet Control Server ===
Duet Control Server version 3.5.0-beta.4 (2023-06-09 10:49:49)
Autopause+ProcessInternally:
>> Macro expansion-reconnect.g: Executing M0
File /opt/dsf/sd/gcodes/TPU - BLVMGNC300 - seal.gcode is selected, cancelled
Autopause:
Buffered code: M0
Buffered codes: 24 bytes total
>> Doing macro expansion-reconnect.g, started by system
Code buffer space: 4072
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 0
Full transfers per second: 42.90, max time between full transfers: 81.7ms, max pin wait times: 69.3ms/7.7ms
Codes per second: 40.54
Maximum length of RX/TX data transfers: 5020/1996

Once this happens, a lot of things occur. I have a expansion-disconnect.g file that I set up to capture the info. I get this:

6/19/2023, 7:20:14 AM 	Error in start-up file expansion-reconnect.g line 8: M562: Board 20 does not have heater 1
6/19/2023, 7:20:14 AM 	Error: M562: Board 20 does not have heater 1
6/19/2023, 7:20:14 AM 	X:150.000 Y:0.000 Z:9.400 E:1465.104 E0:9809.6 Count 30000 30000 7426 Machine 150.000 0.000 9.400 Bed comp -0.118
6/19/2023, 7:20:14 AM 	Printing paused at X237.6 Y133.8 Z4.4
6/19/2023, 7:19:52 AM 	Resume state saved

The contents of the expansion disconnect.g file is:

M25
M114
M562
M122
M122 B20
M0

Once the fault happens, it can't see the heater that's on the toolboard, so it can't be reset for the heat fault that registers because the CAN connection failed.

dc42

@Nuramori thanks, yes the tool board has reset because it ran out of RAM. You've already supplied the GCode file concerned, but please also provide your config.g and any macro files that it calls, so that I can attempt to replicate the issue.

Nuramori

@dc42 Actually, this was different gcode (but far simpler - just two rings out of TPU). Here's the gcode for the file that generated the error....

two rings

config.g

; Configuration file for Duet3 (firmware version 3.4 beta7)
; executed by the firmware on start-up
; 
; --------------- Pin/connection Mapping Legend -------------------
; 0.out0      - Bed heater
; 0.out1
; 0.out2
; 0.out3
; 0.out4      - Radiator cooling fan - WC heatsink
; 0.out5
; 0.out6
; 0.out7
; 0.out8
; 0.out9      - Duet board cooling fan
; 0.out4.tach - Radiator cooling fan rpm wire - WC heatsink
; 0.out5.tach
; 0.out6.tach
; 0.io0.in
; 0.io1.in    - Emergency stop switch
; 0.io2.in    - Y max active low endstop switch
; 0.io3.in
; 0.io4.in
; 0.io5.in
; 0.io6.in
; 0.io7.in
; 0.io8.in
; 0.io0.out
; 0.io1.out
; 0.io2.out
; 0.io3.out
; 0.io4.out
; 0.io5.out
; 0.io6.out
; 0.io7.out
; 0.io8.out
; 0.servo, out10 (only on v0.5, not on v0.6)
; 0.pson
; 0.spi.cs0
; 0.spi.cs1
; 0.spi.cs2
; 0.spi.cs3
; 0.temp0      - Bed Thermistor
; 0.temp1
; 0.temp2
; 0.temp3      - Coolant Temp Sensor
; 0.mcu-temp   - MCU sensor
;
;----------Toolboard 1 (CAN ID#20)
; 20.0         - Accelerometer
; 20.out0      - Hotend Heater
; 20.out1      -
; 20.out2      - Parts cooler on printhead
; 20.out1.tach -
; 20.out2.tach -
; 20.io0.in    - Z probe type to bltouch current - Piezo future
; 20.io1.in    - X min active low endstop switch
; 20.io2.in    -
; 20.io3.in    -
; 20.io0.out   - GPIO port 0 on toolboard io0, servo mode - BLTouch Z-Probe
; 20.temp0     - Extruder Thermistor
; 20.temp1     -
; 20.button0   -
; 20.button1   -
;
; -----------------------------------------------------------------
;
; General preferences
M80 C"!pson"                      ; Declare the Power Pin (inverted for Meanwell)
M80                               ; Turn on the Power
G4 P2000                          ; Hold your Horses.
G90                               ; send absolute coordinates...
M83                               ; ...but relative extruder moves
G21                               ; Set units to Millimeters
M550 P"DUET3"                     ; set printer name
M575 P1 S1 B57600                 ; Set things up for the PanelDue

G4 S6                             ; Wait for toolboard to start
M584 X0.3 Y0.2 Z0.0:0.1:0.4 E20.0 ; set drive mapping to each axis
M669 K1                           ; Select CoreXY mode - New format

; Network
; M552 S1                         ; enable network
; M586 P0 S1                      ; enable HTTP
; M586 P1 S0                      ; disable FTP
; M586 P2 S0                      ; disable Telnet

; Drives
M569 P0.0 S0                    ; physical drive 0 goes backwards - Z Axis Left (1)
M569 P0.1 S0                    ; physical drive 1 goes backwards - Z Axis Right Back (2)
M569 P0.2 S0                    ; physical drive 2 goes backwards - Y Axis
M569 P0.3 S0                    ; physical drive 3 goes backwards - X Axis
M569 P0.4 S0                    ; physical drive 4 goes backwards - Z Axis Right Front (3)
M569 P20.0 S1                   ; physical drive 5 goes forwards - Extruder 1

; Set up three Z-axis location
M671 X-6:362:362 Y186:299:60 S5            ; Pivot Points at 1 - left, 2 - rear right and 3 - front right

M92 X200 Y200 Z800 E1299                  ; set steps per mm
M350 X16 Y16 Z16 E16 I1                    ; configure micro-stepping with interpolation
M566 X600.00 Y600.00 Z100.00 E240.00       ; set maximum instantaneous speed changes (mm/min)
M203 X16000.00 Y16000.00 Z400.00 E1200.00  ; set maximum speeds (mm/min)
M201 X4000.00 Y4000.00 Z400.00 E800.00     ; set accelerations (mm/s^2)
M906 X800 Y800 Z800 E800 I30               ; set motor currents (mA) and motor idle factor in percent*100

; Set up Stealthcop Parameters
; M569 P0.0 V40 H5
; M569 P0.1 V40 H5
; M569 P0.2 V40 H5
; M569 P0.3 V40 H5
; M569 P0.4 V40 H5
; M569 P20.0 V40 H5
; M915 X Y Z E T1

M84 S30                                    ; Set idle timeout

; Axis Limits
M208 X0 Y0 Z0 S1                           ; set axis minima
M208 X315 Y315 Z325 S0                     ; set axis maxima

; Endstops

M574 X1 S1 P"20.io1.in"                    ; X min active high endstop switch
M574 Y2 S1 P"0.io2.in"                     ; Y max active high endstop switch

; Z-Probe - BLTouch - *CURRENT*
M574 Z1 S2                                 ; set endstops controlled by probe
M558 P9 C"^20.io0.in" H5 F2000 T12000      ; set Z probe type to bltouch and the dive height + speeds (bltouch NEEDS pullup)
G31 P100 X0 Y-20 Z3.85                     ; set Z probe trigger value, offset and trigger height - 0.40MM NOZZLE FIXED Kv1

; Configure Heaters and Sensors
M308 S0 P"0.temp0" Y"thermistor" A"Bed Heat" T100000 B3950    ; Configure bed temperature sensor
M950 H0 C"0.out0" T0                                          ; Define heater 0 (bed heater) - bed_heat pin and Temp Sensor 0 
M140 H0                                                       ; Map heated bed to heater 0
M143 H0 P0 S120 A2                                            ; set temperature limit for heater 0 to 120c

M308 S1 P"20.temp0" Y"pt1000" A"Mosquito"                     ; Configure extruder 1 temperature sensor - Mosquito
M950 H1 C"20.out0" T1                                         ; Define heater 1 (hot-end E0) to use the "20.out0" pin and Temp Sensor 1

M143 H1 S280 A2                                               ; set temperature limit for heater 1 to 280C

M308 S2 P"mcu-temp" Y"mcu-temp" A"Duet Board"                 ; Configure MCU sensor
M308 S3 P"temp3" Y"thermistor" T10000 B3988 A"Coolant"        ; Configure coolant sensor
M308 S4 P"drivers" Y"drivers" A"Duet stepper drivers"         ; defines sensor 4 as stepper driver temperature sensor

; Configure Fans
M950 F0 C"20.out2" Q100                 ; Define Fan_0 for use - Parts Cooler on Printhead - 5015 fan
M950 F1 C"0.out9" Q25000                ; Define Fan_1 for use - Duet board cooling fan
M950 F2 C"!0.out4+^0.out4.tach" Q25000  ; Define Fan_1 for use - Radiator cooling - WC heatsink. - PWM fan
M950 S0 C"20.io0.out"                   ; Define GPIO port 0 on IO0, servo mode - BLTouch Z-Probe - *CURRENT*
M950 J0 C"^!0.io1.in"                   ; Input 0 uses 0.io1.in pin, pullup enabled

; Fans
M106 P0 S0                        ; set fan 0. Parts Cooler on Printhead
M106 P1 H2:4 T40:70               ; Set fan 1. Manages Duet board fan.
M106 P2 H3 T35:50                 ; Set fan 1. Manages Radiator fan for water-cooled loop.

;Set PID values        
M307 H0 R0.731 K0.271:0.000 D1.85 E1.35 S1.00 B0         ; disable bang-bang mode for the bed heater and set PWM limit
M307 H1 R2.839 K0.347:0.134 D6.69 E1.35 S1.00 B0 V23.8   ; disable bang-bang mode for the extruder heater and set PWM limit
; Pressure Advance
M572 D0 S0.08                             ; Set pressure advance to offset elasticity

; Dynamic Acceleration Adjustment and Non-linear
; M592 D0 A0.005 B0.0012 L0.2
 M593 P"ei3" F296.5 S0.005

; Set Movement Queue
M595 P100

; Set up Accelerometer
M955 P20.0 I12                      ; Accelerometer on toolboard, oriented counter clockwise 90 Enable for RRF 3.3 Stable

; Tools
M563 P0 S"Printhead" D0 H1 F0       ; define tool 0. Fan 0 operates with an active hot-end
G10 L1 P0 X0 Y0 Z0                  ; set tool 0 axis offsets - - - X15 Y18 - Bed 0,0
G10 P0 R0 S0                        ; set active and standby temperatures to 0C

; Emergency Stop
M574 S1 P"^!0.io1.in"               ; Define Emergency endstop - emergency stop switch condition
M581 P0 S1 T0                       ; Define action to be taken with activation of emergency stop switch

; Miscellaneous
T0                                  ; select first tool

M98 P"homeall.g"                    ; Home X

homing macro that's called:

; homeall.g
; called to home all axes
;
; generated by RepRapFirmware Configuration Tool on Tue Oct 09 2018 13:27:56 GMT+0300
G91                    ; relative positioning

; Home all axis
M98 P"homex.g"         ; Home X
M98 P"homey.g"         ; Home Y
M98 P"homez.g"         ; Home Z

G90                    ; absolute positioning

individual axis macros tha's called in homeall.g:

homex.g

; homex.g
; called to home the X axis
;

G91                  ; relative positioning

; Initiate tuning movement sequence for Stealthcop
; G1 H2 X0.02 F1000
; G4 P50

G1 H2 Z3 F1000       ; Drop Z 3mm
G1 H2 X5 Y5 F4000    ; move X away a little so the switch is not pressed

G1 H1 X-400 F4000    ; move left 400mm, stopping at the endstop
G1 X5 F4000          ; move away for second pass
G4 P500              ; wait 500msec
G1 H3 X-300 F300     ; (second pass)
G1 X15 F4000         ; move away from end
G1 H2 Z-3 F1000      ; Raise z back
G90                  ; back to absolute positioning
G4 P100              ; wait 400ms

homey.g

; homey.g
; called to home the y axis
;

G91                  ; use relative positioning

; Initiate tuning movement sequence for Stealthcop
; G1 H2 Y0.02 F1000
; G4 P50

G1 H2 Z3 F1000       ; Drop Z 3mm
G1 H2 X5 Y5 F4000    ; ensure gantry is clear

G1 H1 Y400 F4000     ; move to the front 400mm, stopping at the endstop
G1 Y-5 F4000         ; move away for second pass
G4 P500              ; wait 500msec
G1 H3 Y400 F300      ; (second pass)
G1 Y-15 F4000        ; move away from end
G1 H2 Z-3 F1000      ; Raise Z back
G90                  ; back to absolute positioning
G4 P100              ; wait 400ms

homez.g

; homez.g
; called to home the Z axis
;
; generated by RepRapFirmware Configuration Tool on Tue Oct 09 2018 13:27:56 GMT+0300
G91				            ; relative positioning

; Initiate tuning movement sequence for Stealthcop
; G1 H2 Z0.02 F1000
; G4 P50

G1 H2 Z10 F1000	            ; Drop Z relative to current position
G90				            ; absolute positioning
G1 X162.5 Y172.5 F16000		; go to first probe point
G30				            ; home Z by probing the bed

; Uncomment the following lines to lift Z after probing
G91				            ; relative positioning
G1 H2 Z10 F1000	            ; lower Z a bit more
G90				            ; absolute positioning

dc42

@Nuramori thanks, I've created this issue https://github.com/Duet3D/RepRapFirmware/issues/876. I'll look into how I can reduce the RAM requirement of your configuration.

For now all I can suggest is that you create fewer devices on the tool board if possible, or switch to an input shaping method that uses fewer acceleration segments such as ei2 or zvdd (the ei3 and zvddd methods use the most).

dc42 created this issue in Duet3D/RepRapFirmware

closed Out-of-memory issues on tool board in firmware 3.5.0-beta.4 #876

Nuramori

@dc42 said in 3.5.0-B4, having random CAN error timeouts:

zvdd

Thanks, I'll switch to ei2 to see if that helps. I looked at the issue info you created - an added note, I am in fact using a raspberry pi 4 with this.

currently all I have on the toolboard are, extruder, fan, heater, and temp sensor.

Nuramori

@dc42 switching to ei2 has led to successful print completions. Thanks for the work around suggestion.

dc42

@Nuramori I'm glad it worked. Please can you post a M122 report after printing that test file, so that I can see how much memory is left.

engikeneer

@dc42 @Nuramori FYI, I have also just had what I think is this issue on my Toolchanger running 3.5.0-B4. It was a ~3 hour overnight print which shows signs of sporadic under extrusion after ~1 hour, then completely stopped extruding at ~ 2 hours, though looks like it continued going to the end of the print judging by the z-height and DWC reporting print complete with no errors.

It seems the printer has regained connection with the toolboard after this, but has lost configuration (heater T0 is offline in DWC and the fans give an error). My guess from the timestamps is that it stopped extruding when the TB reset.

I am using EI3 input shaper so will switch to EI2 and report back

M122

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: QUS4M-U096U-D65J0-40KMD-3W03Z-ZK6H6
Used output buffers: 12 of 40 (40 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 124316 of which 324 recycled
Never used RAM 10628, free system stack 128 words
Tasks: NETWORK(1,ready,175.1%,236) HEAT(3,nWait,0.3%,327) Move(4,nWait,11.7%,244) CanReceiv(6,nWait,0.8%,658) CanSender(5,nWait,0.2%,328) CanClock(7,delaying,0.1%,342) TMC(4,nWait,7.7%,74) MAIN(1,running,92.9%,698) IDLE(0,ready,1.1%,29) AIN(4,delaying,9.2%,266), total 299.1%
Owned mutexes:
=== Platform ===
Last reset 08:45:09 ago, cause: power up
Last software reset at 2023-06-03 10:37, reason: User, Gcodes spinning, available RAM 25980, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 31511040, completed 31511039, timed out 0, errs 0
MCU temperature: min 23.4, current 32.5, max 40.2
Supply voltage: min 24.4, current 24.6, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/156/156, gc cycles 0
Events: 1 queued, 1 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 30, reads 19118, writes 30, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 29, reads 19119, writes 29, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 15, reads 19132, writes 15, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 18, reads 19129, writes 18, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 18, reads 19130, writes 18, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-06-27 07:13:42
Cache data hit count 4294967295
Slowest loop: 1000.22ms; fastest: 0.11ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 4.4ms, write time 2.9ms, max retries 0
=== Move ===
DMs created 83, segments created 39, maxWait 176487ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 214086, completed 214086, hiccups 3825, stepErrors 0, LaErrors 0, Underruns [9, 3, 6], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 470657, received 975846, lost 0, boc 884
Longest wait 3ms for reply type 6053, peak Tx sync delay 49078, free buffers 18 (min 3), ts 157548/156903/0
Tx timeouts 105,0,235,0,0,101 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 5.77ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address f0:08:d1:02:ed:e4
Module reset reason: Power up, Vcc 3.36, flash size 2097152, free heap 33132
WiFi IP address 192.168.1.168
Signal strength -50dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

M122 B22

Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4 (2023-06-08 16:22:30)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 216, free system stack 88 words
Tasks: Move(3,nWait,1.9%,111) HEAT(2,nWait,2.7%,101) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,1.2%,75) CanClock(5,nWait,0.5%,66) ACCEL(3,nWait,0.0%,53) TMC(2,nWait,94.5%,57) MAIN(1,running,607.9%,440) IDLE(0,ready,0.0%,27) AIN(2,delaying,154.2%,142), total 862.8%
Last reset 06:34:17 ago, cause: software
Last software reset time unknown, reason: OutOfMemory, available RAM 4, slot 0
Software reset code 0x01c0 ICSR 0x00000000 SP 0x200054a8 Task Move Freestk 137 ok
Stack: 200057e0 000062df 200057f8 aec314a5 374d23fb 00009f99 200057f8 00007f1f 00000001 4796c700 4796c700 00000000 3f814982 3f814982 20005960 20007268 2000729c 00000000 470f7baa 2f1868ac 2eee1671 2000554c 200072e0 20005108 00000000 00008c91 00000000
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, read errors 0, write errors 1, ifcnt 24, reads 28841, writes 10, timeouts 13, DMA errors 0, CC errors 0, failedOp 0x71, steps req 0 done 1127268
Moves scheduled 45933, completed 45933, in progress 0, hiccups 597, step errors 0, maxPrep 627, maxOverdue 1600706441, maxInc 1600706441, mcErrs 0, gcmErrs 0, ebfmin 0.00, ebfmax 1.00
Peak sync jitter -8/33, peak Rx sync delay 240, resyncs 5/0, no timer interrupt scheduled
VIN voltage: min 24.4, current 24.5, max 24.5
MCU temperature: min 33.5C, current 33.6C, max 57.2C
Last sensors broadcast 0x00000000 found 0 208 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 189524, send timeouts 0, received 343188, lost 66143, free buffers 18, min 0, error reg ff0000
dup 4, oos 74/27/0/39, bm 0, wbm 0, rxMotionDelay 49086, adv -1595093892/1600780883
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 3, other errors 0

Config.g

; Configuration file for Duet 3 Mini 5+ (firmware version 3.3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.3.16 on Sun Apr 02 2023 08:54:48 GMT+0100 (British Summer Time)

; General preferences
M575 P1 S1 B57600                            ; enable support for PanelDue
G90                                          ; send absolute coordinates...
M83                                          ; ...but relative extruder moves
M550 P"Loki"                                 ; set printer name
M669 K1                                      ; select CoreXY mode

; Network
M552 S1                                      ; enable network
M586 P0 S1                                   ; enable HTTP
M586 P1 S0                                   ; disable FTP
M586 P2 S0                                   ; disable Telnet

G4 S4

; Drives
M569 P0.0 S0                                 ; physical drive 0.0 goes backwards (x)
M569 P0.1 S0                                 ; physical drive 0.1 goes backwards (y)
M569 P0.2 S1                                 ; physical drive 0.2 goes forwards (z)
M569 P0.3 S1 D3 V10                                ; physical drive 0.3 goes forwards (pebble wiper)
M569 P0.4 S0                                 ; physical drive 0.4 goes backwards (coupler)
;M569 P20.0 S0                                 ; physical drive 20.0 goes backwards (T0 Motor)
M569 P21.0 S0                                 ; physical drive 20.0 goes backwards (T1 Motor)
M569 P22.0 S0                                 ; physical drive 20.0 goes backwards (T2 Motor)
;M569 P23.0 S0                                 ; physical drive 20.0 goes backwards (T3 Motor)
M584 X0.0 Y0.1 Z0.2 A0.3 C0.4 E22.0:21.0;:20.0:23.0               ; set drive mapping
M350 C8 I0									 ; configure microstepping for coupler
M350 X16 Y16 Z16 A16 E16:16 I1;:16:16 I1                      ; configure microstepping with interpolation
M92 X100 Y100 Z1600 A400 C100 E405:410;:410:410                   ; set steps per mm
M566 X500 Y500 Z100 A50 C2 E1000:1000;:1000:1000           ; set maximum instantaneous speed changes (mm/min)
M203 X20000 Y20000 Z1200 A1500 C5000 E5000:5000;:5000:5000  ; set maximum speeds (mm/min)
M201 X8000 Y8000 Z400 A400 C400 E3000:3000;:3000:3000                 ; set accelerations (mm/s^2)
M906 X1850 Y1850 Z1330 I60                      ; set motor currents (mA) and motor idle factor in percent
M906 A600 C500 E1000:1000  I20 ;:1000:1000                     ; set motor currents (mA) and motor idle factor in percent
M84 S120                                      ; Set idle timeout

; Axis Limits
M98 P"/macros/Set Machine Limits"
;M208 X0 Y0 Z0 S1                             ; set axis minima
;M208 X300 Y210 Z300 S0                       ; set axis maxima

; Endstops
M574 X1 S3                                   ; configure sensorless endstop for low end on X
M574 Y1 S3                                   ; configure sensorless endstop for low end on Y
M574 C1 S3                                   ; configure sensorless endstop for low end on C
M574 A2 S1 P"io1.in"						; configure endstop at low end on A

; Stall Detection
M915 C S5 F0 H200 
M915 X Y S5 F0 H200
M569 P0.3 D3 V10 							; Set Axis A to use stealthchop

;Input Shaping
M593 P"EI3" F44; F42.5


; Z-Probe
M558 P5 C"io5.in" H3 F1000:150 A5 S0.02 T20000     	; set Z probe type to switch and the dive height + speeds
G31 P200 X0 Y0 Z0                            		; set Z probe trigger value, offset and trigger height
M557 X10:290 Y10:190 P20                     		; define mesh grid

; Bed Heater
M308 S0 P"temp0" Y"thermistor" A"Bed" T100000 B4725 C7.06e-8 ; configure sensor 0 as thermistor on pin temp0
M950 H0 C"out0" T0                           ; create bed heater output on out0 and map it to sensor 0
M307 H0 B0 S1.00                             ; disable bang-bang mode for the bed heater and set PWM limit
M140 H0                                      ; map heated bed to heater 0
M143 H0 S150                                 ; set temperature limit for heater 0 to 120C

; Pebble Wiper
M950 S0 C"io3.out" Q300			; Define Pebble Wiper Servo

; Tool 0
M308 S1 P"22.temp0" Y"thermistor" A"T0" T100000 B4725 C7.06e-8 ; Set thermistor
M950 H1 C"22.out0" T1                                         ; Extruder 0 heater
M143 H1 S300                                                 ; Set temperature limit for heater 1 to 300C
M950 F1 C"22.out2"
M950 F2 C"22.out1"
M106 P1 S255 H1 T70             ; T0 HE
M106 P2 S0 B0.25                     ; T0 PCF
M563 P0 S"T0 0.4mm" D0 H1 F2         ; Define tool 0
G10 P0 X0 Y0 Z0                         ; Reset tool 0 axis offsets
G10 P0 R0 S0                            ; Reset initial tool 0 active and standby temperatures to 0C

; Tool 1
M308 S2 P"21.temp0" Y"thermistor" A"T1" T100000 B4725 C7.06e-8 ; Set thermistor
M950 H2 C"21.out0" T2                                         ; Extruder 1 heater
M143 H2 S300    ; Set temperature limit for heater 2 to 300C    
M950 F3 C"21.out2"
M950 F4 C"21.out1"
M106 P3 S255 H2 T70             ; T1 HE fan thermostatic to heater 2
M106 P4 S0 B0.25                      ; T1 PCF
M563 P1 S"T1 0.4mm" D1 H2 F4         ; Define tool 1
G10 P1 X0 Y0 Z0                         ; Reset tool 1 axis offsets
G10 P1 R0 S0                            ; Reset initial tool 1 active and standby temperatures to 0C

;tool offsets
G10 P0 X23 Y41 Z-5.6				; T0 
G10 P1 X23 Y41 Z-5.9			; T1

; Pressure Advance
M572 D0 S0.03 						; pressure advance T0
M572 D1 S0.03						; pressure advance T1

; Firmware Retraction
M207 P0 S0.4 F3000
M207 P1 S0.4 F3000


; Pid Tuning
M307 H0 R1.607 K0.253:0.000 D3.06 E1.35 S1.00 B0
M307 H1 R2.556 K0.326:0.519 D6.03 E1.35 S1.00 B0 V24.1
M307 H2 R1.813 K0.362:0.007 D6.08 E1.35 S1.00 B0 V24.4

;Extra Sensors
M308 S5 Y"mcu-temp" A"MCU"  

; Fans & LED's
M950 F0 C"out_6"				; Define enclosure LED pin
M950 F9 C"out_5" Q12000				; Define board cooling fan
M106 P9 L50 X200 H5 T30:80				; Set board cooling fans to come on when MCU temp is above 40C

jay_s_uk

@engikeneer from what i understand, although the firmware now know's that, its only a warning and nothing is done with it https://docs.duet3d.com/en/User_manual/RepRapFirmware/Events
it may be worth implementing a macro to pause the print so at least you can possibly save it.
The only issue I could envisage is that the configuration won't get resent to the toolboard without restarting or sending M98 P"config.g" so you may need to have another macro containing just the toolboard config info so you can get it reconfigured before restarting the print.

engikeneer

@jay_s_uk Hi, yeah I do need to set up the events system to control how it handles things. However the default behaviour should at least report it in the console and that print did not seem to have anything reported.

However, I have just had repeated the same issue with using EI2 shaper, got the out-of-memory reset and it reported it in the console. It got slightly further through the print this time (so it's probably not a specific print issue). It also had potentially more of the random under-extrusions - this may or may not be a red herring and I intend to swap the hemera on that tool to see if it is mechanical.

I will try again with no shaper at all. Is there anything else I could try/provide to help narrow this down?

Data with EI2 shaper:

M122

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: QUS4M-U096U-D65J0-40KMD-3W03Z-ZK6H6
Used output buffers: 3 of 40 (40 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 123924 of which 12 recycled
Never used RAM 11548, free system stack 124 words
Tasks: NETWORK(2,nWait,29.4%,205) HEAT(3,nWait,0.1%,327) Move(4,nWait,4.8%,244) CanReceiv(6,nWait,0.2%,658) CanSender(5,nWait,0.1%,328) CanClock(7,delaying,0.0%,342) TMC(4,nWait,1.5%,74) MAIN(1,running,61.9%,698) IDLE(0,ready,0.4%,29) AIN(4,delaying,1.7%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 03:11:17 ago, cause: software
Last software reset at 2023-06-27 07:33, reason: User, Gcodes spinning, available RAM 10628, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00487000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 11478083, completed 11478082, timed out 0, errs 0
MCU temperature: min 32.5, current 34.6, max 40.2
Supply voltage: min 24.4, current 24.6, max 24.6, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/156/156, gc cycles 0
Events: 3 queued, 3 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 60, reads 10777, writes 15, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 60, reads 10776, writes 16, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 31, reads 10788, writes 3, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 36, read errors 0, write errors 0, ifcnt 37, reads 10789, writes 3, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 0, ifcnt 41, reads 10788, writes 4, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-06-27 10:45:15
Cache data hit count 4294967295
Slowest loop: 1000.15ms; fastest: 0.10ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 9.6ms, write time 2.1ms, max retries 0
=== Move ===
DMs created 83, segments created 30, maxWait 80617ms, bed compensation in use: mesh, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 165321, completed 165321, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 30, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 245248, received 421404, lost 0, boc 1584
Longest wait 271ms for reply type 6029, peak Tx sync delay 48880, free buffers 18 (min 3), ts 57129/55779/0
Tx timeouts 528,0,609,0,0,266 last cancelled message type 52 dest 22
=== Network ===
Slowest loop: 217.54ms; fastest: 0.06ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address f0:08:d1:02:ed:e4
Module reset reason: Power up, Vcc 3.35, flash size 2097152, free heap 33056
WiFi IP address 192.168.1.168
Signal strength -44dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 3 0 0 0 0 0 0 0

M122 B22

	M122 B22
Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4 (2023-06-08 16:22:30)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 1272, free system stack 88 words
Tasks: Move(3,nWait,0.4%,111) HEAT(2,nWait,0.1%,101) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.1%,75) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,nWait,3.0%,57) MAIN(1,running,91.4%,438) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.9%,142), total 100.0%
Last reset 00:37:41 ago, cause: software
Last software reset at 2023-06-27 09:59, reason: OutOfMemory, available RAM 4, slot 1
Software reset code 0x01c0 ICSR 0x00000000 SP 0x200054a8 Task Move Freestk 137 ok
Stack: 200057e0 000062df 200057f8 ae3a643e 36f08716 00009f99 200057f8 00007dd5 0001e926 3f62a79c 20005108 20007538 3d926a20 3d926a20 2dfcf290 200074c0 200074f4 00000000 470550b8 2ef478c0 2ef478c0 2000554c 20007538 20005108 00000000 00008c91 00000000
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, read errors 0, write errors 1, ifcnt 50, reads 15453, writes 11, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 867312
Moves scheduled 32394, completed 32394, in progress 0, hiccups 264, step errors 0, maxPrep 435, maxOverdue 2031906521, maxInc 2030421997, mcErrs 0, gcmErrs 0, ebfmin 0.00, ebfmax 1.00
Peak sync jitter -6/36, peak Rx sync delay 233, resyncs 11/1, no timer interrupt scheduled
VIN voltage: min 24.4, current 24.5, max 24.5
MCU temperature: min 40.1C, current 40.3C, max 56.9C
Last sensors broadcast 0x00000000 found 0 59 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 18111, send timeouts 2, received 60794, lost 0, free buffers 18, min 17, error reg ff0000
Last cancelled message type 4525 dest 0
dup 9, oos 284/99/0/242, bm 0, wbm 0, rxMotionDelay 367, adv -2031906298/222232
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 3, other errors 0

jay_s_uk

@engikeneer I don't think so, its just waiting on @dc42 to look at the issue.
I've been printing mostly using RP2040 based toolboards recently and extrusion has been fine there so I don't think there are currently any more extrusion issues, I think its more down to lack of memory than anything else.
Maybe try a different shaper again sych as ZVD or ZVDD.
I think also providing the file you are printing as well as your config.g, any start files and daemon.g will help dc42 investigate the issue

dc42

@engikeneer if it got further with EI2 that suggests that it is indeed the extra RAM used to implement input shaping that is causing it to run out of memory. You could try ZVD or MZV, both of which use less RAM than EI2.

I'll work on freeing up RAM for the next build of the tool board firmware.

engikeneer

@dc42 Thanks for the feedback. I can confirm that the same print managed to complete with all input shaping turned off. M122 and M122 B22 for that below

M122

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: QUS4M-U096U-D65J0-40KMD-3W03Z-ZK6H6
Used output buffers: 3 of 40 (40 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 123996 of which 216 recycled
Never used RAM 11464, free system stack 122 words
Tasks: NETWORK(2,nWait,188.1%,203) HEAT(3,nWait,0.4%,329) Move(4,nWait,32.5%,244) CanReceiv(6,nWait,1.2%,646) CanSender(5,nWait,0.5%,328) CanClock(7,delaying,0.1%,342) TMC(4,nWait,9.1%,74) MAIN(1,running,413.9%,704) IDLE(0,ready,3.7%,29) AIN(4,delaying,10.7%,266), total 660.2%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 03:27:55 ago, cause: software
Last software reset at 2023-06-27 10:48, reason: User, Gcodes spinning, available RAM 11548, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 12476123, completed 12476122, timed out 0, errs 0
MCU temperature: min 34.4, current 34.6, max 40.2
Supply voltage: min 24.4, current 24.6, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/156/156, gc cycles 0
Events: 23 queued, 23 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 93, reads 274, writes 30, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 93, reads 273, writes 30, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 48, reads 288, writes 15, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 56, reads 285, writes 18, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 63, reads 285, writes 19, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-06-27 14:15:58
Cache data hit count 4294967295
Slowest loop: 1518.61ms; fastest: 0.09ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 10.3ms, write time 2.1ms, max retries 0
=== Move ===
DMs created 83, segments created 22, maxWait 28487ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 213813, completed 213813, hiccups 0, stepErrors 0, LaErrors 0, Underruns [9, 218, 4], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 299313, received 476178, lost 0, boc 5514
Longest wait 518ms for reply type 6033, peak Tx sync delay 49047, free buffers 18 (min 3), ts 62379/56331/0
Tx timeouts 3407,0,3644,1,0,1956 last cancelled message type 4514 dest 127
=== Network ===
Slowest loop: 217.85ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address f0:08:d1:02:ed:e4
Module reset reason: Power up, Vcc 3.35, flash size 2097152, free heap 39612
WiFi IP address 192.168.1.168
Signal strength -48dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

M122 B22

M122 B22
Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4 (2023-06-08 16:22:30)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 772, free system stack 88 words
Tasks: Move(3,nWait,0.4%,111) HEAT(2,nWait,0.5%,57) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.1%,75) CanClock(5,nWait,0.0%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,3.0%,57) MAIN(1,running,91.1%,344) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.9%,142), total 100.0%
Last reset 03:28:25 ago, cause: software
Last software reset at 2023-06-27 09:59, reason: OutOfMemory, available RAM 4, slot 1
Software reset code 0x01c0 ICSR 0x00000000 SP 0x200054a8 Task Move Freestk 137 ok
Stack: 200057e0 000062df 200057f8 ae3a643e 36f08716 00009f99 200057f8 00007dd5 0001e926 3f62a79c 20005108 20007538 3d926a20 3d926a20 2dfcf290 200074c0 200074f4 00000000 470550b8 2ef478c0 2ef478c0 2000554c 20007538 20005108 00000000 00008c91 00000000
Driver 0: pos 0, 405.0 steps/mm, standstill, SG min 0, read errors 0, write errors 0, ifcnt 66, reads 54732, writes 1, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 1412618
Moves scheduled 151605, completed 151605, in progress 0, hiccups 535, step errors 0, maxPrep 639, maxOverdue 1611293, maxInc 629772, mcErrs 0, gcmErrs 0, ebfmin -1.00, ebfmax 1.00
Peak sync jitter -15/43, peak Rx sync delay 285, resyncs 101/1, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.5, max 24.5
MCU temperature: min 36.4C, current 39.7C, max 57.5C
Last sensors broadcast 0x00000002 found 1 202 ticks ago, 0 ordering errs, loop time 1
CAN messages queued 68358, send timeouts 151, received 94836, lost 0, free buffers 18, min 17, error reg ff0000
Last cancelled message type 4514 dest 127
dup 22, oos 557/212/0/757, bm 0, wbm 0, rxMotionDelay 444, adv -1611092/222205
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 0, other errors 0

There were two new things that I did spot which I am not sure if they are related.... First is I got a bunch of timeout errors on my two toolboards for their temp sensors (see below from console). I managed to watch it and it did look like it lost CAN connection/sync from what I could see of the LED's. You can see a bunch of timeouts in the CAN part of M122.

The second thing is that at certain points in the print, the head motion seemed to get very jerky. The print is three objects - one printed fine and two have horrific under extrusion. The times I saw it, the jerky motion was on the poorly printed parts. I wonder if the CAN timeouts meant that the movement queue got interrupted so the toolhead had to stop? Might just have been luck that one part came out fine?

6/27/2023, 1:40:33 PM 	Error: Expansion board 21 stopped sending status
Error: Expansion board 22 stopped sending status
Debug from 22: temp timeout on sensor 1
Debug from 22: temp timeout on sensor 1
temp timeout on sensor 1
temp timeout o
Debug from 22: n sensor 1
temp timeout on sensor 1
Debug from 21: temp timeout on sensor 2
Debug from 21: temp timeout on sensor 2
temp timeout on sensor 2
temp timeout o
Debug from 21: n sensor 2
temp timeout on sensor 2
temp timeout on sensor 2

Print file below. rename back to a .zip and extract to get the gcode - was slightly too big....
Low Poly Tall candle holder v2x3_TC.zip.txt

I have no daemon.g and no start.g. I can provide the homing files etc but there's not much interesting in them - I haven't really needed to use much meta commands yet.

Edit: Here is an image of the print. Back object is okay, front two are not happy! They should all be the same

dc42

@engikeneer I've put a new build of the tool board firmware with more free RAM at https://www.dropbox.com/sh/c4aatfdsejjjab0/AACGuZ4qUBvZz49zxErkhmnWa?dl=0. Please try it to see if this out-of-memory issue is resolved.

engikeneer

@dc42 Thanks, that appears to have fixed it and the print completed fine and thge jerky motion I saw was gone. M122 and M122 B22 below if you're intersted.

The under extrusion remains but is much less severe. I got couple of Expansion board 22 stopped sending status (and board 21) during the print. It seems fine printing from T1 (board 21), so I am guessing that this is due to a CAN wiring issue or some interference on T0 (board 22) as that is the first board in the chain. I will make up some new CAN cables in the next week or so, but either way I am convinced it is not a Beta FW issue

M122

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: QUS4M-U096U-D65J0-40KMD-3W03Z-ZK6H6
Used output buffers: 3 of 40 (40 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 123924 of which 216 recycled
Never used RAM 11152, free system stack 128 words
Tasks: NETWORK(2,nWait,49.4%,205) HEAT(3,nWait,0.1%,327) Move(4,nWait,8.4%,244) CanReceiv(6,nWait,0.3%,672) CanSender(5,nWait,0.1%,328) CanClock(7,delaying,0.0%,342) TMC(4,delaying,2.4%,74) MAIN(1,running,35.5%,698) IDLE(0,ready,0.8%,29) AIN(4,delaying,2.9%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 04:35:03 ago, cause: software
Last software reset at 2023-07-01 09:00, reason: User, Gcodes spinning, available RAM 14832, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 16505513, completed 16505513, timed out 0, errs 0
MCU temperature: min 29.2, current 34.0, max 38.9
Supply voltage: min 24.4, current 24.6, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/312/312, gc cycles 0
Events: 2 queued, 2 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 57, reads 15760, writes 42, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 57, reads 15759, writes 42, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 33, reads 15786, writes 15, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 34, reads 15783, writes 18, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 36, reads 15780, writes 21, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-07-01 13:35:54
Cache data hit count 4294967295
Slowest loop: 999.94ms; fastest: 0.10ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 10.3ms, write time 8.0ms, max retries 0
=== Move ===
DMs created 83, segments created 38, maxWait 218705ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 213781, completed 213781, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 3, 1], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 377421, received 659870, lost 0, boc 578
Longest wait 480ms for reply type 6029, peak Tx sync delay 47658, free buffers 18 (min 3), ts 82520/82158/0
Tx timeouts 63,0,89,0,0,53 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 215.44ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address f0:08:d1:02:ed:e4
Module reset reason: Power up, Vcc 3.36, flash size 2097152, free heap 39628
WiFi IP address 192.168.1.168
Signal strength -53dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

M122 B22

Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4+ (2023-06-30 15:40:06)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 2760, free system stack 88 words
Tasks: Move(3,nWait,1.5%,71) HEAT(2,nWait,1.7%,37) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.4%,75) CanClock(5,nWait,0.1%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,10.4%,57) MAIN(1,running,69.4%,322) IDLE(0,ready,0.0%,27) AIN(2,delaying,16.7%,114), total 100.0%
Last reset 04:34:51 ago, cause: software
Last software reset at 2023-06-27 09:59, reason: OutOfMemory, available RAM 4, slot 1
Software reset code 0x01c0 ICSR 0x00000000 SP 0x200054a8 Task Move Freestk 137 ok
Stack: 200057e0 000062df 200057f8 ae3a643e 36f08716 00009f99 200057f8 00007dd5 0001e926 3f62a79c 20005108 20007538 3d926a20 3d926a20 2dfcf290 200074c0 200074f4 00000000 470550b8 2ef478c0 2ef478c0 2000554c 20007538 20005108 00000000 00008c91 00000000
Driver 0: pos 0, 405.0 steps/mm, standstill, SG min 0, read errors 1, write errors 0, ifcnt 39, reads 28857, writes 2, timeouts 3, DMA errors 0, CC errors 0, failedOp 0x6f, steps req 0 done 11981257
Moves scheduled 220948, completed 220948, in progress 0, hiccups 2525, segs 51, step errors 0, maxPrep 1497, maxOverdue 764587, maxInc 239236, mcErrs 0, gcmErrs 0, ebfmin -1.00 max 1.00
Peak sync jitter -10/29, peak Rx sync delay 393, resyncs 3/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.5, max 24.5
MCU temperature: min 32.4C, current 39.1C, max 51.3C
Last sensors broadcast 0x00000002 found 1 8 ticks ago, 0 ordering errs, loop time 1
CAN messages queued 325377, send timeouts 6, received 436534, lost 0, free buffers 18, min 17, error reg ff0000
Last cancelled message type 4514 dest 127
dup 6, oos 712/200/0/149, bm 0, wbm 0, rxMotionDelay 584, adv -763616/100219
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 0, other errors 0