Duet3 1LC resets during print job - does not pause

[mct82]

IDEX machine running RRF 3.4 on a Duet3 6HC + (2) 1LC tool boards. The machine will start and run a job for minutes to hours prior to ceasing to extrude and registering the following error:

Board 20 does not have input handle 1000
Failed to enable endstops

The build job continues without pausing or aborting. The M122 from the most recent case are as follows:

M122B20
Diagnostics for board 20:
Duet TOOL1LC rev 1.0 or earlier firmware version 3.4.0 (2022-03-15 09:00:04)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 4064, free system stack 45 words
Tasks: Move(notifyWait,0.0%,91) HEAT(notifyWait,0.1%,115) CanAsync(notifyWait,0.0%,65) CanRecv(notifyWait,0.0%,74) CanClock(notifyWait,0.0%,65) TMC(delaying,3.0%,57) MAIN(running,92.0%,441) IDLE(ready,0.0%,40) AIN(delaying,4.9%,142), total 100.0%
Last reset 00:47:23 ago, cause: power up
Last software reset data not available
Driver 0: pos 3071110, 80.0 steps/mm,standstill, SG min 2, read errors 0, write errors 0, ifcnt 10, reads 62986, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 6772, completed 6772, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 4/13, peak Rx sync delay 212, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.4
MCU temperature: min 74.0C, current 76.2C, max 90.2C
Last sensors broadcast 0x00000000 found 0 177 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 9424, send timeouts 0, received 15276, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: none
I2C bus errors 4493, naks 4493, other errors 4493
4/24/2022, 1:00:07 PM 	

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.0 (2022-03-15 18:57:24) running on Duet 3 MB6HC v1.01 or later (SBC mode)
Board ID: 08DJM-956L2-G43S8-6JKD0-3S46L-9U2LD
Used output buffers: 1 of 40 (15 max)
=== RTOS ===
Static ram: 151000
Dynamic ram: 68444 of which 0 recycled
Never used RAM 127748, free system stack 114 words
Tasks: SBC(ready,15.1%,458) HEAT(notifyWait,0.4%,321) Move(notifyWait,20.2%,248) CanReceiv(notifyWait,1.1%,772) CanSender(notifyWait,0.2%,346) CanClock(delaying,0.1%,339) TMC(notifyWait,62.4%,58) MAIN(running,0.3%,923) IDLE(ready,0.1%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 28:21:37 ago, cause: power up
Last software reset at 2022-04-23 00:42, reason: User, GCodes spinning, available RAM 127892, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
Step timer max interval 136
MCU temperature: min 25.1, current 50.7, max 51.2
Supply voltage: min 0.3, current 32.2, max 32.7, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 0.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/890/890, gc cycles 115
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 728, reads 19389, writes 26 timeouts 0
Driver 1: standstill, SG min 0, mspos 776, reads 19393, writes 22 timeouts 0
Driver 2: standstill, SG min 0, mspos 216, reads 19389, writes 26 timeouts 0
Driver 3: standstill, SG min 0, mspos 56, reads 19389, writes 26 timeouts 0
Driver 4: standstill, SG min 0, mspos 328, reads 19389, writes 26 timeouts 0
Driver 5: standstill, SG min 0, mspos 152, reads 19389, writes 26 timeouts 0
Date/time: 2022-04-24 18:00:03
Slowest loop: 82.55ms; fastest: 0.03ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 26, maxWait 4422119ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 14459, completed 14459, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 14], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters 3 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.2
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon* is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== Filament sensors ===
Extruder 0: no data received
Extruder 1: no data received
=== CAN ===
Messages queued 1514237, received 4970374, lost 0, boc 0
Longest wait 5ms for reply type 6024, peak Tx sync delay 604, free buffers 50 (min 43), ts 510490/510489/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 4, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 39972/39972
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0, IAP RAM available 0x2b858
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server v3.4.0
File /opt/dsf/sd/gcodes/scaledJ0035ACD_NEST_0.25_PLA_1d11h33m.gcode is selected, paused
Last invalid JSON response:
{"key":"move","flags":"d99vn","result":{"axes":[{"acceleration":1500.0,"babystep":0,"current":2000,"drivers":["0.0"],"homed":true,"jerk":250.0,"letter":"X","machinePosition":111.355,"max":445.00,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-82.00,"minProbed":false,"percentCurrent":100,"percentStstCurrent":71,"speed":20000.0,"stepsPerMm":80.00,"userPosition":268.947,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]},{"acceleration":1500.0,"babystep":0,"current":2000,"drivers":["0.4","0.5"],"homed":true,"jerk":250.0,"letter":"Y","machinePosition":338.503,"max":450.00,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-43.00,"minProbed":false,"percentCurrent":100,"percentStstCurrent":71,"speed":20000.0,"stepsPerMm":80.00,"userPosition":43.981,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]},{"acceleration":150.0,"babystep":-1.900,"current":1500,"drivers":["0.2","0.3"],"homed":true,"jerk":25.0,"letter":"Z","machinePosition":-1.500,"max":620.00,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-7.30,"minProbed":false,"percentCurrent":100,"percentStstCurrent":71,"speed":1500.0,"stepsPerMm":1600.00,"userPosition":0.350,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]},{"acceleration":1500.0,"babystep":0,"current":2000,"drivers":["0.1"],"homed":true,"jerk":250.0,"letter":"U","machinePosition":533.000,"max":533.00,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":5.00,"minProbed":false,"percentCurrent":100,"percentStstCurrent":71,"speed":20000.0,"stepsPerMm":80.00,"userPosition":533.000,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]}],"calibration":{"final":{"deviation":0.000,"mean":0.000},"initial":{"deviation":0.006,"mean":-1.590},"numFactors":2},"compensation":{"fadeHeight":5.0,"file":null,"liveGrid":null,"meshDeviation":null,"probeGrid":{"axes":["X","Y"],"maxs":[425.0,325.0],"mins":[25.0,125.0],"radius":-1.0,"spacings":[100.0,100.0]},"skew":{"compensateXY":true,"tanXY":0,"tanXZ":0,"tanYZ":0},"type":"none"},"currentMove":{"acceleration":500.0,"deceleration":500.0,"laserPwm":null,"requestedSpeed":37.5,"topSpeed":37.5},"extruders":[{"acceleration":10000.0,"current":1000,"driver":"20.0","factor":1.000,"filament":"","jerk":200.0,"microstepping":{"interpolated":true,"value":16},"nonlinear":{"a":0,"b":0,"upperLimit":0.20},"percentCurrent":100,"percentStstCurrent":71,"position":330.7,"pressureAdvance":0,"rawPosition":358.5,"speed":2400.0,"stepsPerMm":675.00},{"acceleration":10000.0,"current":1000,"driver":"21.0","factor":1.000,"filament":"","jerk":200.0,"microstepping":{"interpolated":true,"value":16},"nonlinear":{"a":0,"b":0,"upperLimit":0.20},"percentCurrent":100,"percentStstCurrent":71,"position":0,"pressureAdvance":0,"rawPosition":0,"speed":2400.0,"stepsPerMm":675.00}],"idle":{"factor":0.3,"timeout":300.0},"kinematics":{"forwardMatrix":[[1.000,0,0,0],[0,1.000,0,0],[0,0,1.000,0],[0,0,0,1.000]],"inverseMatrix":[[1.000,0,0,0],[0,1.000,0,0],[0,0,1.000,0],[0,0,0,1.000]],"name":"cartesian","tiltCorrection":{"correctionFactor":1.0,"lastCorrections":[-1.601,-1.579],"maxCorrection":10.0,"screwPitch":0.50,"screwX":[-150.0,600.0],"screwY":[225.0,225.0]},"segmentation":{"minSegLength":0.01,"segmentsPerSec":1.0}},"limitAxes":true,"noMovesBeforeHoming":true,"printingAcceleration":500.0,"queue":[{"gracePeriod":0.010,"length":60},{"gracePeriod":0.010,"length":5}],"rotation":{},"shaping":{"amplitudes":[],"damping":0.10,"durations":[],"frequency":40.00,"minAcceleration":0.0,"type":"none"},"speedFactor":1.00,"travelAcceleration":2000.0,"virtualEPos":0,"workplaceNumber":0}}

Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 0
Full transfers per second: 39.06, max time between full transfers: 1155.7ms, max pin wait times: 107.2ms/18.8ms
Codes per second: 6.06
Maximum length of RX/TX data transfers: 3884/1736

It sounds like this issue has been related to heater errors causing the 1LC to reboot. There is currently a magnetic filament monitor configured on the 1LC but sensing is disabled, not sure if that could be causing the issue.

Let me know what else is needed to diagnose.

[dc42]

@mct82 according to the software reset data, the last reset was 47 minutes before you ran the M122 command and was caused by power up. Did you power up the printer 47 minutes ago, or did the print go wrong 47 minutes ago?

[mct82]

@dc42 The printer was unattended and the reset was 47 minutes ago. In that case it had been running for only 15-20 minutes before the error.

There is another M122 in that code block that shows the 6HC being powered up for >28hours.

[dc42]

@mct82 the data indicates that the reset was caused by power up. So the microcontroller on the tool board lost power. This might mean that you have a bad connection in the power wiring to the tool board, or possibly that there was a short between 12V or 5V on the tool board and ground.

[mct82]

@dc42 So I made no changes to the wiring, but commented the M591’s out of the config and restarted. I was able to complete a 36hr build w/o interruption.

With that knowledge I will double check the wiring to the MFM. I’m wondering if a “sensor error” on the MFM could be causing a reset on the 1LC? Or maybe a wiring problem is causing both the sensor error and/or the 1LC resets.

[Phaedrux]

I think this issue may already be under investigation with the filament monitors.

[dc42]

@mct82 said in Duet3 1LC resets during print job - does not pause:

@dc42 So I made no changes to the wiring, but commented the M591’s out of the config and restarted. I was able to complete a 36hr build w/o interruption.

With that knowledge I will double check the wiring to the MFM. I’m wondering if a “sensor error” on the MFM could be causing a reset on the 1LC? Or maybe a wiring problem is causing both the sensor error and/or the 1LC resets.

If a software error caused the 1LC to reset, then the "Last reset reason" would show as "Software", not "Power up".