@dc42 So I made no changes to the wiring, but commented the M591’s out of the config and restarted. I was able to complete a 36hr build w/o interruption.
With that knowledge I will double check the wiring to the MFM. I’m wondering if a “sensor error” on the MFM could be causing a reset on the 1LC? Or maybe a wiring problem is causing both the sensor error and/or the 1LC resets.
@dc42 The printer was unattended and the reset was 47 minutes ago. In that case it had been running for only 15-20 minutes before the error.
There is another M122 in that code block that shows the 6HC being powered up for >28hours.
IDEX machine running RRF 3.4 on a Duet3 6HC + (2) 1LC tool boards. The machine will start and run a job for minutes to hours prior to ceasing to extrude and registering the following error:
Board 20 does not have input handle 1000
Failed to enable endstops
The build job continues without pausing or aborting. The M122 from the most recent case are as follows:
M122B20
Diagnostics for board 20:
Duet TOOL1LC rev 1.0 or earlier firmware version 3.4.0 (2022-03-15 09:00:04)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 4064, free system stack 45 words
Tasks: Move(notifyWait,0.0%,91) HEAT(notifyWait,0.1%,115) CanAsync(notifyWait,0.0%,65) CanRecv(notifyWait,0.0%,74) CanClock(notifyWait,0.0%,65) TMC(delaying,3.0%,57) MAIN(running,92.0%,441) IDLE(ready,0.0%,40) AIN(delaying,4.9%,142), total 100.0%
Last reset 00:47:23 ago, cause: power up
Last software reset data not available
Driver 0: pos 3071110, 80.0 steps/mm,standstill, SG min 2, read errors 0, write errors 0, ifcnt 10, reads 62986, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 6772, completed 6772, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 4/13, peak Rx sync delay 212, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.4
MCU temperature: min 74.0C, current 76.2C, max 90.2C
Last sensors broadcast 0x00000000 found 0 177 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 9424, send timeouts 0, received 15276, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: none
I2C bus errors 4493, naks 4493, other errors 4493
4/24/2022, 1:00:07 PM
M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.0 (2022-03-15 18:57:24) running on Duet 3 MB6HC v1.01 or later (SBC mode)
Board ID: 08DJM-956L2-G43S8-6JKD0-3S46L-9U2LD
Used output buffers: 1 of 40 (15 max)
=== RTOS ===
Static ram: 151000
Dynamic ram: 68444 of which 0 recycled
Never used RAM 127748, free system stack 114 words
Tasks: SBC(ready,15.1%,458) HEAT(notifyWait,0.4%,321) Move(notifyWait,20.2%,248) CanReceiv(notifyWait,1.1%,772) CanSender(notifyWait,0.2%,346) CanClock(delaying,0.1%,339) TMC(notifyWait,62.4%,58) MAIN(running,0.3%,923) IDLE(ready,0.1%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 28:21:37 ago, cause: power up
Last software reset at 2022-04-23 00:42, reason: User, GCodes spinning, available RAM 127892, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
Step timer max interval 136
MCU temperature: min 25.1, current 50.7, max 51.2
Supply voltage: min 0.3, current 32.2, max 32.7, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 0.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/890/890, gc cycles 115
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 728, reads 19389, writes 26 timeouts 0
Driver 1: standstill, SG min 0, mspos 776, reads 19393, writes 22 timeouts 0
Driver 2: standstill, SG min 0, mspos 216, reads 19389, writes 26 timeouts 0
Driver 3: standstill, SG min 0, mspos 56, reads 19389, writes 26 timeouts 0
Driver 4: standstill, SG min 0, mspos 328, reads 19389, writes 26 timeouts 0
Driver 5: standstill, SG min 0, mspos 152, reads 19389, writes 26 timeouts 0
Date/time: 2022-04-24 18:00:03
Slowest loop: 82.55ms; fastest: 0.03ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 26, maxWait 4422119ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 14459, completed 14459, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 14], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters 3 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.2
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon* is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== Filament sensors ===
Extruder 0: no data received
Extruder 1: no data received
=== CAN ===
Messages queued 1514237, received 4970374, lost 0, boc 0
Longest wait 5ms for reply type 6024, peak Tx sync delay 604, free buffers 50 (min 43), ts 510490/510489/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 4, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 39972/39972
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0, IAP RAM available 0x2b858
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server v3.4.0
File /opt/dsf/sd/gcodes/scaledJ0035ACD_NEST_0.25_PLA_1d11h33m.gcode is selected, paused
Last invalid JSON response:
{"key":"move","flags":"d99vn","result":{"axes":[{"acceleration":1500.0,"babystep":0,"current":2000,"drivers":["0.0"],"homed":true,"jerk":250.0,"letter":"X","machinePosition":111.355,"max":445.00,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-82.00,"minProbed":false,"percentCurrent":100,"percentStstCurrent":71,"speed":20000.0,"stepsPerMm":80.00,"userPosition":268.947,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]},{"acceleration":1500.0,"babystep":0,"current":2000,"drivers":["0.4","0.5"],"homed":true,"jerk":250.0,"letter":"Y","machinePosition":338.503,"max":450.00,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-43.00,"minProbed":false,"percentCurrent":100,"percentStstCurrent":71,"speed":20000.0,"stepsPerMm":80.00,"userPosition":43.981,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]},{"acceleration":150.0,"babystep":-1.900,"current":1500,"drivers":["0.2","0.3"],"homed":true,"jerk":25.0,"letter":"Z","machinePosition":-1.500,"max":620.00,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-7.30,"minProbed":false,"percentCurrent":100,"percentStstCurrent":71,"speed":1500.0,"stepsPerMm":1600.00,"userPosition":0.350,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]},{"acceleration":1500.0,"babystep":0,"current":2000,"drivers":["0.1"],"homed":true,"jerk":250.0,"letter":"U","machinePosition":533.000,"max":533.00,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":5.00,"minProbed":false,"percentCurrent":100,"percentStstCurrent":71,"speed":20000.0,"stepsPerMm":80.00,"userPosition":533.000,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]}],"calibration":{"final":{"deviation":0.000,"mean":0.000},"initial":{"deviation":0.006,"mean":-1.590},"numFactors":2},"compensation":{"fadeHeight":5.0,"file":null,"liveGrid":null,"meshDeviation":null,"probeGrid":{"axes":["X","Y"],"maxs":[425.0,325.0],"mins":[25.0,125.0],"radius":-1.0,"spacings":[100.0,100.0]},"skew":{"compensateXY":true,"tanXY":0,"tanXZ":0,"tanYZ":0},"type":"none"},"currentMove":{"acceleration":500.0,"deceleration":500.0,"laserPwm":null,"requestedSpeed":37.5,"topSpeed":37.5},"extruders":[{"acceleration":10000.0,"current":1000,"driver":"20.0","factor":1.000,"filament":"","jerk":200.0,"microstepping":{"interpolated":true,"value":16},"nonlinear":{"a":0,"b":0,"upperLimit":0.20},"percentCurrent":100,"percentStstCurrent":71,"position":330.7,"pressureAdvance":0,"rawPosition":358.5,"speed":2400.0,"stepsPerMm":675.00},{"acceleration":10000.0,"current":1000,"driver":"21.0","factor":1.000,"filament":"","jerk":200.0,"microstepping":{"interpolated":true,"value":16},"nonlinear":{"a":0,"b":0,"upperLimit":0.20},"percentCurrent":100,"percentStstCurrent":71,"position":0,"pressureAdvance":0,"rawPosition":0,"speed":2400.0,"stepsPerMm":675.00}],"idle":{"factor":0.3,"timeout":300.0},"kinematics":{"forwardMatrix":[[1.000,0,0,0],[0,1.000,0,0],[0,0,1.000,0],[0,0,0,1.000]],"inverseMatrix":[[1.000,0,0,0],[0,1.000,0,0],[0,0,1.000,0],[0,0,0,1.000]],"name":"cartesian","tiltCorrection":{"correctionFactor":1.0,"lastCorrections":[-1.601,-1.579],"maxCorrection":10.0,"screwPitch":0.50,"screwX":[-150.0,600.0],"screwY":[225.0,225.0]},"segmentation":{"minSegLength":0.01,"segmentsPerSec":1.0}},"limitAxes":true,"noMovesBeforeHoming":true,"printingAcceleration":500.0,"queue":[{"gracePeriod":0.010,"length":60},{"gracePeriod":0.010,"length":5}],"rotation":{},"shaping":{"amplitudes":[],"damping":0.10,"durations":[],"frequency":40.00,"minAcceleration":0.0,"type":"none"},"speedFactor":1.00,"travelAcceleration":2000.0,"virtualEPos":0,"workplaceNumber":0}}
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 0
Full transfers per second: 39.06, max time between full transfers: 1155.7ms, max pin wait times: 107.2ms/18.8ms
Codes per second: 6.06
Maximum length of RX/TX data transfers: 3884/1736
It sounds like this issue has been related to heater errors causing the 1LC to reboot. There is currently a magnetic filament monitor configured on the 1LC but sensing is disabled, not sure if that could be causing the issue.
Let me know what else is needed to diagnose.
@medicusdkfz Historically I had my sensors floating "inline" like your pictures show, and I also battled these spurious faults and exaggerated min/max values. At some point I finally took the time to design housings/brackets to positively affix the monitors to a rigid structure (extruder, carriage, chassis, etc) and the faults disappeared. I am guessing it had something to do with twisting of the filament/ptfe tube, and/or lash in the filament conveyance path. It seems like however small the error was, eventually it would stack up and release at just the wrong time to trigger a fault.
Any thoughts? Should I move this to a different forum? FYI, this is a SBC install.
Here is a gcode snip from a recent print:
;FLAVOR:RepRap
M73 P0
M73 R1144
;TIME:68644
;Filament used: 187.431m, 24.1528m
;Layer height: 0.25
;MINX:15.415
;MINY:28.434
;MINZ:0.4
;MAXX:324.707
;MAXY:423.334
;MAXZ:107.4
;POSTPROCESSED
;Generated with Cura_SteamEngine 4.10.0
;
M140 S70
G10 P0 S230.0 R180
G10 P1 S255 R200
M98 P"slicer_start.g"
T0
M83 ;relative extrusion mode
G1 F1800 E-1.2
;LAYER_COUNT:428
;LAYER:0
M107
M104 T1 S200
G1 F100 Z0.9 ;------MODIFIED------
G0 F9000 X109.529 Y47.02 Z0.9
;TYPE:SKIRT
G1 F1500 Z0.4
G1 F1800 E1.2
G1 X109.997 Y46.023 E0.1218
G1 X110.311 Y45.384 E0.07874
G1 X110.616 Y44.782 E0.07463
G1 X110.727 Y44.577 E0.02578
G1 X111.149 Y43.852 E0.09277
G1 X111.565 Y43.162 E0.0891
G1 X111.93 Y42.573 E0.07663
G1 X112.061 Y42.376 E0.02616
G1 X112.494 Y41.76 E0.08327
G1 X113.044 Y41.002 E0.10357
I flagged the line that was causing missed Z steps. Originally it was at F1500 and would have run fine previously. The following line didn't seem to cause problems in this program, so I left it at F9000. On other programs where the distance from the parked position back to the print was further, it would cause lost steps in X as well. The distance-dependence is what makes me think it is related to acceleration. The configuration is a Dual X carriage idex w/ toolheads parked at the extents when not in use.
Similar lost X step issues would occur later in the program as the tool changes came back into play, but I didn't include any of that because its a lot of macros to digest. We can look at that later if necessary.
Config.g
; Configuration file for Duet 3 (firmware version 3)
; executed by the firmware on start-up
;
;
;
; General preferences
G90 ; send absolute coordinates...
M83 ; ...but relative extruder moves
M550 P"Aon-M2" ; set printer name
M555 P0 ; set RRF emulation mode
G4 S1 ; wait for expansion boards to start
;
;
; Network
;M551 P"aonm2" ; set password
;M552 S1 ; enable network
;M586 P0 S1 ; enable HTTP
;M586 P1 S0 ; disable FTP
;M586 P2 S1 ; enable Telnet
;
;
; Drives
M569 P0 S0 D2 ; physical drive 0, X, X_L
M569 P1 S0 D2 ; physical drive 1, U, X_R
M569 P2 S1 D2 ; physical drive 2, Z, Z_L
M569 P3 S1 D2 ; physical drive 3, Z, Z_R
M569 P4 S1 D2 ; physical drive 4, Y, Y_R
M569 P5 S0 D2 ; physical drive 5, Y, Y_L
;M569 P1.0 S0 D3 V135 ; physical drive 6, n/a
;M569 P1.1 S1 D3 V135 ; physical drive 7, n/a
;M569 P1.2 S1 D3 V135 ; physical drive 8, n/a
M569 P20.0 S1 D2 ; physical drive 9, E0
M569 P21.0 S0 D2 ; physical drive 10, E1
M584 X0 Y4:5 Z2:3 U1 E20.0:21.0 ; set drive mapping
M350 X16 Y16 Z16 U16 E16 I1 ; x16 microstepping w interpo
M92 X80 Y80 Z1600 U80 E675 ; set steps per mm
M203 X20000 Y20000 Z1500 U20000 E2400 ; set max speed (mm/min)
M201 X1500 Y1500 Z150 U1500 E10000 ; set max accel (mm/s^2)
M566 X250 Y250 Z25 U250 E200 P1 ; set max jerk (mm/min)
M906 X2000 Y2000 Z1500 U2000 E1000 I30 ; set motor currents (mA) and motor idle factor in per cent
M84 S300 ; Set idle timeout
M669 K0 S1.0 T0.010 ; set kinematics and segementation
M204 P1000 T2000 ; set print and travel max accels
;
;
; Axis Limits
M208 X-82 Y-43 Z-7.3 U5 S1 ; set axis minima
M208 X445 Y450 Z620 U533 S0 ; set axis maxima
M564 S1 H1 ; limit movement to homed axes within limits
;
;
; Endstops
M574 X1 S1 P"^20.io2.in" ; configure x endstop
M574 Y1 S1 P"^io4.in+^io5.in" ; configure dual y endstops
;M574 Y1 S1 P"^io5.in" ; configure single y endstop
;M574 V1 S1 P"^io4.in" ; configure single v endstop
;M574 Z1 S1 P"^io2.in" ; configure single z endstop
M574 Z1 S1 P"^io2.in+^io3.in" ; configure dual z endstops
M574 U2 S1 P"^21.io2.in" ; configure u endstop
;
;
; Z-Probes
M558 K0 P8 C"20.io0.in" B1 F250 H10 T10000 ; set Z probe 0 input
G31 K0 P500 X0 Y0 Z-3.025 ; set Z probe 0 params
M290 R0 Z-1.45 ; set default babysteps
;
;M558 K1 P8 C"21.io0.in" B1 F250 H15 T10000 ; set Z probe 1 input
;G31 K1 P500 X0 Y0 Z-4.144 ; set Z probe 1 params
;
;
; Heaters
M308 S0 P"temp0" Y"thermistor" T100000 B4138 C0 A"Bed" ; sensor0 = bed thermistor
M950 H0 C"out0" T0 ; create heater H0
M307 H0 R0.507 C310.5 D3.58 S1.00 V0.0 ; H0 params
M140 H0 ; H0 = bed heater
M143 H0 S200 ; set max bed temp
;
M308 S1 P"20.temp0" Y"pt1000" A"E0" ; sensor 1 PT1000
M950 H1 C"20.out0" T1 ; nozzle heater output H1
M307 H1 A787 C300 D6.0 S1.0 V0.0 B0 ; H1 params
M143 H1 P0 S350 A0 C0 ; H1 max temp
;
M308 S2 P"21.temp0" Y"pt1000" A"E1" ; sensor 2 PT1000
M950 H2 C"21.out0" T2 ; nozzle heater H2
M307 H2 A787 C300 D6.0 S1.0 V0.0 B0 ; H2 params
M143 H2 P0 S350 A0 C0 ; H2 max temp
;
M308 S3 P"temp3" Y"pt1000" A"Chmbr" ; chamber thermistor
M950 H3 C"out7" T3 ; chamber heater H3
M141 H3 ; H3 = chamber
M307 H3 A107.6 C953.2 D45.4 S1.0 V0.0 B0 ; H3 params
;
;
; Fans
M950 F0 C"20.out1" Q500 ; fan0, T0 blower
M106 P0 S0.0 B0.5 H-1 C"T0 Fan" X0.66 ;
;
M950 F1 C"21.out1" Q500 ; fan1, T1 blower
M106 P1 S0.0 B0.5 H-1 C"T1 Fan" X0.66 ;
;
;M950 F3 C"out3" Q500 ; fan2, bed blower
;M106 P3 S0.0 B0.5 H-1 C"Bed Fan" ;
;
M950 F4 C"out8" ; fan3, chmbr htr fan
;M106 P4 S0.0 H3 T35 C"Chmbr Fan" ; thermostatic chamber fan
M106 P4 S0.0 H-1 C"Chmbr Fan" ; manual chamber fan
;
;
; Tools
M563 P0 S"LEFT" D0 H1 F0 ; define tool 0
G10 P0 X0 Y0 Z0 ; T0 offsets
G10 P0 R0 S0 ; T0 temps
;
M563 P1 S"RIGHT" D1 H2 F1 X3 ; define tool 1
G10 P1 X0 Y-0.35 U0.3 Z-.18 ; T1 offsets
G10 P1 R0 S0 ; T1 temps
;
;M563 P2 S"DUPL" D0:1 H1:2 F0:1 X0:3 ; T0 uses both extruders and hotends, maps X to both X and U, and uses both print fans
;G10 P2 X0 Y0 U-225 ; T0 offsets
;G10 P2 S0 R0 ; T0 temps
;M567 P2 E1:1 ; T0 mix ratio 100% on both extruders
;
;
; Filament Sensors
M591 D0 P3 C"20.io1.in" R25:250 A0 E10.0 L25.2 S1 ; T0 mag sensor
M591 D1 P3 C"21.io1.in" R25:250 A0 E10.0 L25.2 S1 ; T1 mag sensor
;
;
; Other Settings
M570 H1 P60 T15 S0 ; relaxed H1 fault detection
M570 H2 P60 T15 S0 ; relaxed H2 fault detection
M671 X-150:600 Y225:225 S10 ; 2-point bed leveling
M557 X25:425 Y125:325 P5:3
;M557 X25:425 Y25:425 P7:7 ; define G29 mesh grid
M376 H5 ; bed comp taper
M572 D0 S0.00 ; E0 pressure advance
M572 D1 S0.00 ; E1 pressure advance
;M579 X1.0020 Y1.0018 ; fine scaling adjustment, 70°C PEI bed reference
;M579 X1.0100 Y1.0100 ; Scaling for J9CHB
M911 S28.0 R50.0 P"M913 X0 Y0 U0 G91 M83 G1 Z3 E-5 F1000" ; enable power failure recovery
;
;
; Miscellaneous
M929 P"BuildLog.txt" S3 ; enable build log
M501 ; restore settings from config-override.g
M144 P0 S0 ; set bed to standby
T1 P0
T0 P0 ; select T0, no macros
T-1 P0
M122 B0
M122 B0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3 (2021-06-15 21:45:47) running on Duet 3 MB6HC v1.01 or later (SBC mode)
Board ID: 08DJM-956L2-G43S8-6JKD0-3S46L-9U2LD
Used output buffers: 1 of 40 (14 max)
=== RTOS ===
Static ram: 150904
Dynamic ram: 64168 of which 0 recycled
Never used RAM 136264, free system stack 116 words
Tasks: SBC(ready,54.7%,314) HEAT(delaying,2.5%,325) Move(notifyWait,8.1%,250) CanReceiv(notifyWait,4.1%,774) CanSender(notifyWait,0.2%,362) CanClock(delaying,0.9%,339) TMC(notifyWait,105.5%,59) MAIN(running,41.6%,922) IDLE(ready,0.0%,29), total 217.7%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 209:44:14 ago, cause: power up
Last software reset at 2021-07-31 06:48, reason: User, none spinning, available RAM 136448, slot 2
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0440f000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
Step timer max interval 243
MCU temperature: min 41.0, current 47.7, max 51.6
Supply voltage: min 2.8, current 32.2, max 32.4, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 0.2, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/656/656, gc cycles 924
Driver 0: position -6559, standstill, reads 33362, writes 28 timeouts 0, SG min/max 0/892
Driver 1: position 35200, standstill, reads 33362, writes 28 timeouts 0, SG min/max 0/1023
Driver 2: position 373898, standstill, reads 33358, writes 32 timeouts 0, SG min/max 0/1023
Driver 3: position 42640, standstill, reads 33358, writes 32 timeouts 0, SG min/max 0/1023
Driver 4: position 0, standstill, reads 33363, writes 28 timeouts 0, SG min/max 0/1023
Driver 5: position 0, standstill, reads 33363, writes 28 timeouts 0, SG min/max 0/1023
Date/time: 2021-08-10 14:35:50
Slowest loop: 308.61ms; fastest: 0.02ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 393880453ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 99178, completed moves 99178, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 351, 21], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = 3 -1 -1 -1
Heater 0 is on, I-accum = 0.3
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon* is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause* is idle in state(s) 0
Code queue is empty.
=== Filament sensors ===
Extruder 0: no data received
Extruder 1: no data received
=== CAN ===
Messages queued 8019658, received 18880625, lost 0, longest wait 4ms for reply type 6024, peak Tx sync delay 509, free buffers 49 (min 29), ts 3775271/3775270/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
State: 4, failed transfers: 1, checksum errors: 0
Last transfer: 1ms ago
RX/TX seq numbers: 59944/59944
SPI underruns 0, overruns 0
Disconnects: 0, timeouts: 0, IAP RAM available 0x2c810
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.3.0
Code buffer space: 4096
Configured SPI speed: 8000000Hz
Full transfers per second: 37.47, max wait times: 193.3ms/0.0ms
Codes per second: 1.84
Maximum length of RX/TX data transfers: 3492/1736
M122 B1
M122 B1
Diagnostics for board 1:
Duet EXP3HC firmware version 3.3 (2021-06-15 16:12:41)
Bootloader ID: not available
Never used RAM 159104, free system stack 4378 words
Tasks: Move(notifyWait,0.1%,152) HEAT(delaying,1.3%,104) CanAsync(notifyWait,0.0%,69) CanRecv(notifyWait,2.3%,80) CanClock(notifyWait,1.2%,71) TMC(notifyWait,53.5%,63) MAIN(running,73.9%,414) IDLE(ready,0.0%,39) AIN(delaying,85.5%,263), total 217.8%
Last reset 209:44:09 ago, cause: power up
Last software reset data not available
Driver 0: position 0, 80.0 steps/mm, standstill, reads 53017, writes 11 timeouts 0, SG min/max 0/0, steps req 0 done 0
Driver 1: position 0, 80.0 steps/mm, standstill, reads 53018, writes 11 timeouts 0, SG min/max 0/0, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm, standstill, reads 53018, writes 11 timeouts 0, SG min/max 0/0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -6/17, peak Rx sync delay 184, resyncs 0/0, no step interrupt scheduled
VIN: 32.3V, V12: 12.3V
MCU temperature: min 37.1C, current 41.4C, max 44.8C
Ticks since heat task active 87, ADC conversions started 755049269, completed 755049267, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 91 ticks ago, loop time 0
CAN messages queued 25, send timeouts 0, received 12835712, lost 0, free buffers 37, min 37, error reg 20000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
M122 B20
M122 B20
Diagnostics for board 20:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Bootloader ID: not available
Never used RAM 3240, free system stack 2717 words
Tasks: Move(notifyWait,9.0%,91) HEAT(delaying,26.1%,99) CanAsync(notifyWait,0.0%,55) CanRecv(notifyWait,5.1%,74) CanClock(notifyWait,2.9%,65) TMC(notifyWait,88.9%,57) MAIN(running,22.7%,350) IDLE(ready,0.0%,41) AIN(delaying,63.3%,142), total 218.1%
Last reset 209:43:56 ago, cause: power up
Last software reset data not available
Driver 0: position 140308470, 675.0 steps/mm, standstill, SG min/max 0/156, read errors 0, write errors 0, ifcnt 16, reads 10413, writes 16, timeouts 3, DMA errors 0, failedOp 0x01, steps req 156499290 done 156499290
Moves scheduled 1108076, completed 1108076, in progress 0, hiccups 0, step errors 0, maxPrep 663, maxOverdue 222718, maxInc 222016, mcErrs 0, gcmErrs 0
Peak sync jitter -23/15, peak Rx sync delay 235, resyncs 0/0, no step interrupt scheduled
VIN: 24.4V
MCU temperature: min 37.5C, current 41.6C, max 72.1C
Ticks since heat task active 174, ADC conversions started 754718586, completed 754718585, timed out 0, errs 0
Last sensors broadcast 0x00000002 found 1 179 ticks ago, loop time 0
CAN messages queued 9439861, send timeouts 0, received 10925124, lost 0, free buffers 37, min 35, error reg 40000
dup 0, oos 0/0/0/0, bm 93, wbm 222718, rxMotionDelay 774, adv 21747/74662
Accelerometer detected: no
I2C bus errors 4619, naks 4619, other errors 4619
=== Filament sensors ===
Interrupt 4 to 36us, poll 8 to 4622us
Driver 0: pos 166.29, errs: frame 358 parity 0 ovrun 0 pol 0 ovdue 0
M122 B21
M122 B21
Diagnostics for board 21:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Bootloader ID: not available
Never used RAM 3272, free system stack 2717 words
Tasks: Move(notifyWait,1.1%,91) HEAT(delaying,26.3%,99) CanAsync(notifyWait,0.0%,57) CanRecv(notifyWait,3.8%,74) CanClock(notifyWait,3.0%,65) TMC(notifyWait,89.5%,57) MAIN(running,34.5%,352) IDLE(ready,0.0%,41) AIN(delaying,60.7%,142), total 218.9%
Last reset 209:43:25 ago, cause: power up
Last software reset data not available
Driver 0: position 16856633, 675.0 steps/mm, standstill, SG min/max 0/270, read errors 0, write errors 0, ifcnt 14, reads 689, writes 14, timeouts 0, DMA errors 0, steps req 20189513 done 20189513
Moves scheduled 112703, completed 112703, in progress 0, hiccups 0, step errors 0, maxPrep 660, maxOverdue 222728, maxInc 222633, mcErrs 0, gcmErrs 0
Peak sync jitter 0/15, peak Rx sync delay 228, resyncs 0/0, no step interrupt scheduled
VIN: 24.0V
MCU temperature: min 37.2C, current 41.4C, max 65.9C
Ticks since heat task active 117, ADC conversions started 754737760, completed 754737759, timed out 0, errs 0
Last sensors broadcast 0x00000004 found 1 123 ticks ago, loop time 0
CAN messages queued 9439102, send timeouts 0, received 9928935, lost 0, free buffers 37, min 36, error reg 120000
dup 0, oos 0/0/0/0, bm 83, wbm 222728, rxMotionDelay 605, adv 22085/74658
Accelerometer detected: no
I2C bus errors 4619, naks 4619, other errors 4619
=== Filament sensors ===
Interrupt 4 to 36us, poll 8 to 2860us
Driver 0: pos 318.16, errs: frame 74 parity 0 ovrun 0 pol 0 ovdue 0
Duet3 w/ 1LC tool boards in IDEX config. After update from 3.2.2 to 3.3 (release), I have started missing X/Y/Z steps in the initial moves following a tool change. I have been able to avoid the step loss issue by setting the Z feedrate to ~100mm/min and X/Y to ~1000mm/min for the first moves. If the feeds are left at the normal values (that worked in 3.2.2), the affected steppers bang/buzz like they are being driven at too high of an acceleration. Motors are already being driven with 2A at 32V. Have not had any issues previously with high speeds or accelerations.
After the initial movement, the system appears to operate normally, respecting all the max accel and jerk settings
Let me know what you’d like me to provide to help diagnose.
@Torin I'm not a programmer and I have never used the Go language, so I don't think you want me mucking up your code. I just don't have time to learn a new language right now, so I will have to wait for someone who knows what they're doing to do it properly.
I will mention that if I were to modify this tool in any way, it would be to pull the message content out so it can be edited w/o re-compiling. Perhaps read in a message template for each status (print, pause, finish), or a single template with multiple sections. It would be really powerful to be able to grab additional fields from the OM to include in (or conditionally modify) push notifications.
@Torin said in notifications upon start/pause/finish of print:
Also added re-notify feature.
The re-notify feature does not appear to be working under windows. I receive the first notification for any new status, but no subsequent updates after status changes. I see a reference to time.Now().Unix() in the main.go source, could this be part of the issue?
@Torin Well it works perfectly when set up properly 
So to change the content of the attachment, I just need to update these sections and recompile?
func sendNotification(status rrStatusType, fileinfo rrFileinfoType) {
messageTitle := `Your friendly 3d printer status`
messageText := fmt.Sprintf(`Status: %v, print file name: %v, print time: %v`, returnStatus(status.Status), fileinfo.FileName, niceTime(fileinfo.PrintDuration))
if status.TimesLeft.File != 0 && status.Status == `P` {
messageText = fmt.Sprintf(`, print time left: %v`, niceTime(int(status.TimesLeft.File)))
}
and
if viper.GetBool(`notifications.slack.enabled`) &&
viper.Get(`notifications.slack.webhookurl`) != `` &&
contains(returnStatus(status.Status), viper.GetStringSlice(`notifications.slack.when`)) {
go func(messageText string, messageTitle string) {
log.Printf("Sending slack message\n")
message := slackRequestBody{
Attachments: []attachment{{
Title: messageTitle,
Text: messageText,
Color: "good",
},
},
}
sendSlackNotification(message)
}(messageTitle, messageText)
}
Thank you for putting this together!
@Torin Thanks for the quick update!
I documented a little of my setup to help with troubleshooting, maybe something has changed on Slack's end?
Here is my webhook setup:
And here is the next action in the workflow:
This is the result posted to the channel:
Is there a way to post to a channel from a webhook w/o using the workflow tool? Maybe that's something I missed....
@Torin Can you provide a little guidance on the Slack integration setup?
I have created a workflow and successfully posted to a channel, however I'm not able to determine which variable names to use to extract information from the web request. Slack doesn't provide many details for troubleshooting.
The workflow completes successfully with a single variable named "text", but this appears to be empty. Do I need to reconfigure the contents of notifications.go to eliminate the nested json (slack docs say they aren't supported for variables)?
if viper.GetBool(`notifications.slack.enabled`) && viper.Get(`notifications.slack.webhookurl`) != `` && contains(returnStatus(status.Status), viper.GetStringSlice(`notifications.pushover.when`)) {
go func(messageText string, messageTitle string) {
log.Printf("Sending slack message\n")
message := slackRequestBody{
Attachments: []attachment{{
Title: messageTitle,
Text: messageText,
Color: "good",
},
},
}
sendSlackNotification(message)
}(messageTitle, messageText)
}
Also seems like "viper.GetStringSlice" should refer to notifications.slack.when, correct?
The heater faults appear to have been resolved by switching back to known-good PT1000 sensors. I guess I will look into wtf is going on with these new RTDs.
Since the failure-to-pause issue is already a known issue, I'll mark this as resolved. Thanks
@dc42 Negative. It's been difficult to catch, but I did see a trace on the temp graph that just started rolling down from a definite fault point. Prior to that the trace was within the normal +/-1C.
I don't think 275C should be anywhere near the limit for these PT1000s, but these are both new and from the same batch.
The issue has just recurred, this time running no chamber heat and lower bed temp. The max MCU temps are lower, but the hotend heaters' status was both "FAULT". They did not fault simultaneously based on the print failures (~1hr apart).
The diagnostics look like this:
2/18/2021, 3:05:15 AM M122 B21
Diagnostics for board 21:
Duet TOOL1LC firmware version 3.2 (2021-01-05)
Bootloader ID: not available
Never used RAM 3648, free system stack 24 words
HEAT 86 CanAsync 89 CanRecv 83 TMC 54 MAIN 218 AIN 64
Last reset 04:59:48 ago, cause: software
Last software reset data not available
Driver 0: position 26542684, 675.0 steps/mm, standstill, SG min/max 0/176, read errors 0, write errors 1, ifcnt 120, reads 15580, writes 17, timeouts 0, DMA errors 0
Moves scheduled 182892, completed 182892, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 24.0V
MCU temperature: min 46.8C, current 47.5C, max 59.2C
Ticks since heat task active 102, ADC conversions started 17916430, completed 17916429, timed out 0
Last sensors broadcast 0x00000004 found 1 105 ticks ago, loop time 0
CAN messages queued 224907, send timeouts 0, received 416810, lost 0, free buffers 36
=== Filament sensors ===
Interrupt 5 to 33us, poll 8 to 764us
Driver 0: pos 30.94, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
2/18/2021, 3:04:54 AM M122 B20
Diagnostics for board 20:
Duet TOOL1LC firmware version 3.2 (2021-01-05)
Bootloader ID: not available
Never used RAM 3648, free system stack 22 words
HEAT 86 CanAsync 89 CanRecv 83 TMC 54 MAIN 216 AIN 64
Last reset 04:59:28 ago, cause: software
Last software reset data not available
Driver 0: position 26552134, 675.0 steps/mm, standstill, SG min/max 0/114, read errors 0, write errors 0, ifcnt 118, reads 3506, writes 1, timeouts 0, DMA errors 0
Moves scheduled 182901, completed 182901, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 24.5V
MCU temperature: min 47.7C, current 47.7C, max 61.1C
Ticks since heat task active 41, ADC conversions started 17896490, completed 17896489, timed out 0
Last sensors broadcast 0x00000002 found 1 44 ticks ago, loop time 0
CAN messages queued 145963, send timeouts 0, received 290355, lost 0, free buffers 36
=== Filament sensors ===
Interrupt 5 to 33us, poll 9 to 685us
Driver 0: pos 348.05, errs: frame 59 parity 0 ovrun 0 pol 0 ovdue 0
M122 B0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2 running on Duet 3 MB6HC v1.01 or later (SBC mode)
Board ID: 08DJM-956L2-G43S8-6JKD0-3S46L-9U2LD
Used output buffers: 1 of 40 (15 max)
=== RTOS ===
Static ram: 149788
Dynamic ram: 64972 of which 164 recycled
Never used RAM 143908, free system stack 122 words
Tasks: Linux(ready,77) HEAT(blocked,271) CanReceiv(blocked,809) CanSender(blocked,335) CanClock(blocked,352) TMC(blocked,19) MAIN(running,671) IDLE(ready,19)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 05:03:23 ago, cause: software
Last software reset at 2021-02-18 03:05, reason: User, GCodes spinning, available RAM 143948, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 43.1, current 43.2, max 43.3
Supply voltage: min 32.2, current 32.2, max 32.2, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0
Driver 0: position -6560, standstill, reads 14561, writes 0 timeouts 0, SG min/max not available
Driver 1: position -3440, standstill, reads 14561, writes 0 timeouts 0, SG min/max not available
Driver 2: position 5767, standstill, reads 14561, writes 0 timeouts 0, SG min/max not available
Driver 3: position 42640, standstill, reads 14561, writes 0 timeouts 0, SG min/max not available
Driver 4: position 0, standstill, reads 14561, writes 0 timeouts 0, SG min/max not available
Driver 5: position 0, standstill, reads 14561, writes 0 timeouts 0, SG min/max not available
Date/time: 2021-02-18 08:08:47
Slowest loop: 0.28ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 185922, completed moves 185922, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = 3 -1 -1 -1
Heater 0 is on, I-accum = 0.2
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause* is idle in state(s) 0
Code queue is empty.
=== Filament sensors ===
Extruder 0: no data received
Extruder 1: no data received
=== CAN ===
Messages queued 11, send timeouts 0, received 68, lost 0, longest wait 0ms for reply type 0, free buffers 48
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 1ms ago
RX/TX seq numbers: 42120/42120
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x2c8a8
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.2.0
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 35.97
Maximum length of RX/TX data transfers: 3812/1684
Sorry for the confusion, the hotend(s) are going into a heater fault condition...as in: press the heater name on I/F, wait for warning timer, reset. The lack of a msg in the console is partly why I found this issue concerning.
This and previous jobs have been running on T2, which is a duplication mode tool using both carriages. Config.g attached.
I have a Duet3+SBC installed in an Aon M2 running RRF3.2 with 1LC toolboards on each X carriage. Up until this point I had been printing lower temp materials (240-260C), however I'm running some material now at 275C and the 1LCs have been triggering heater faults a couple hours into the builds.
The 1LCs are necessarily in the heated chamber, but we had been running the chamber at 50C and bed around 70C without issues. The step up from 260 to 275 seems to have been the tipping point.
Here are the diags from the toolboards, at this stage board 21 had a heater fault:
2/17/2021, 8:58:47 PM M122 B20
Diagnostics for board 20:
Duet TOOL1LC firmware version 3.2 (2021-01-05)
Bootloader ID: not available
Never used RAM 3600, free system stack 22 words
HEAT 86 CanAsync 85 CanRecv 83 TMC 54 MAIN 218 AIN 64
Last reset 12:18:54 ago, cause: software
Last software reset data not available
Driver 0: position 56823563, 675.0 steps/mm, standstill, SG min/max 0/148, read errors 0, write errors 1, ifcnt 99, reads 16102, writes 21, timeouts 0, DMA errors 0
Moves scheduled 403829, completed 403829, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 23.8V
MCU temperature: min 40.2C, current 65.0C, max 73.8C
Ticks since heat task active 132, ADC conversions started 44158069, completed 44158067, timed out 0
Last sensors broadcast 0x00000002 found 1 135 ticks ago, loop time 0
CAN messages queued 554665, send timeouts 0, received 980476, lost 0, free buffers 36
=== Filament sensors ===
Interrupt 5 to 44us, poll 9 to 793us
Driver 0: pos 118.12, errs: frame 12 parity 0 ovrun 0 pol 0 ovdue 0
2/17/2021, 8:57:53 PM M122 B21
Diagnostics for board 21:
Duet TOOL1LC firmware version 3.2 (2021-01-05)
Bootloader ID: not available
Never used RAM 3600, free system stack 22 words
HEAT 86 CanAsync 85 CanRecv 83 TMC 54 MAIN 208 AIN 64
Last reset 12:18:00 ago, cause: software
Last software reset data not available
Driver 0: position 56943713, 675.0 steps/mm, standstill, SG min/max 0/124, read errors 0, write errors 1, ifcnt 102, reads 54571, writes 22, timeouts 0, DMA errors 0
Moves scheduled 403818, completed 403818, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 23.9V
MCU temperature: min 38.5C, current 62.4C, max 69.9C
Ticks since heat task active 249, ADC conversions started 44104626, completed 44104625, timed out 0
Last sensors broadcast 0x00000004 found 1 3 ticks ago, loop time 0
CAN messages queued 553745, send timeouts 0, received 979676, lost 0, free buffers 36
=== Filament sensors ===
Interrupt 5 to 36us, poll 8 to 813us
Driver 0: pos 356.13, errs: frame 6 parity 0 ovrun 0 pol 0 ovdue 0
Note that none of the heater faults ever trigger an error on the DWC or pause the print job. I see those MCU temps are creeping up towards the 80+ range that could potentially be an issue, is that the likely cause?
The hotend setup is 24V/60W heaters w/ PT1000 RTDs. I have liquid cooling on the carriages and a spare fan output on the 1LCs, I was just hoping to not need to engineer the board cooling just yet. Both space and time are tight. I went through all the manual heater tuning prior to this last build to make sure there weren't some temp excursions happening that I wasn't seeing on the graph, the tuning seems ok and stable.
Let me know if overheating is the likely culprit, or if something else is amiss. It certainly seems like the desired behavior would be for a 1LC heater fault to pause the build.
Is it possible to control a heater connected via 1LC using a PT100 connected to a Duet3 6HC w/ daughterboard?