Duet 6HC randomly crashing

agmeadows

My Duet 6HC has been randomly crashing. It ran great for a couple of years but about 6 months ago started crashing randomly. I though it was my smart outlet I used to turn it off and on. I removed that and it seemed to resolve the issue. Unfortunately, it started crashing again. Here is what I have done recently:

Updated the firmware to 3.4.5
Enclosed the printer so the chamber can be heated (passively by bed only)

Here is the latest diagnostic output:

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.5 (2022-11-30 19:35:23) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-956BA-NA3TJ-6JTD8-3SJ6J-1UA2U
Used output buffers: 3 of 40 (36 max)
=== RTOS ===
Static ram: 152760
Dynamic ram: 97584 of which 76 recycled
Never used RAM 100252, free system stack 204 words
Tasks: NETWORK(ready,29.3%,230) ETHERNET(notifyWait,0.1%,443) HEAT(notifyWait,0.0%,322) Move(notifyWait,0.0%,351) CanReceiv(notifyWait,0.0%,944) CanSender(notifyWait,0.0%,336) CanClock(delaying,0.0%,333) TMC(notifyWait,8.0%,91) MAIN(running,62.6%,925) IDLE(ready,0.0%,30), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:12:40 ago, cause: software
Last software reset at 2023-04-03 10:46, reason: MemoryProtectionFault mmarValid daccViol, FilamentSensors spinning, available RAM 95728, slot 1
Software reset code 0x416d HFSR 0x00000000 CFSR 0x00000082 ICSR 0x00427804 BFAR 0x00000018 SP 0x2041b5d0 Task MAIN Freestk 1724 ok
Stack: 2041b648 00000000 00000001 2041b6a0 00000000 00471029 00471632 210f0000 00000000 2042474c 2042aa60 00472589 00000000 00488ee7 00000000 00000000 204305a8 2041b648 000000fb 0047295d 00000005 204305a8 204305a8 00472941 204305a8 004716fb 00000000
Error status: 0x00
Step timer max interval 128
MCU temperature: min 37.1, current 40.1, max 40.3
Supply voltage: min 23.8, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 24, reads 26789, writes 15 timeouts 0
Driver 1: standstill, SG min 0, mspos 600, reads 26789, writes 15 timeouts 0
Driver 2: standstill, SG min 0, mspos 152, reads 26790, writes 15 timeouts 0
Driver 3: standstill, SG min 0, mspos 328, reads 26790, writes 15 timeouts 0
Driver 4: standstill, SG min 0, mspos 376, reads 26790, writes 15 timeouts 0
Driver 5: standstill, SG min 0, mspos 264, reads 26790, writes 15 timeouts 0
Date/time: 2023-04-03 10:59:36
Slowest loop: 3.98ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.4ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== Filament sensors ===
Extruder 0: pos 2.84, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
=== CAN ===
Messages queued 6842, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 50), ts 3802/0/0
Tx timeouts 0,0,3801,0,0,3039 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 21.81ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 3 of 8
= Ethernet =
State: active
Error counts: 0 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

I used M98 to verify the config.g file and there are warnings for max bed temp but no errors. Here is my config.g file:

M552 P0.0.0.0 S1
G21								;Work in millimetres
G90								;Send absolute coordinates...
M83								;...but relative extruder moves

; Stepper configuration
M569 P0 S1 D3					;Drive 0 X / Rear
M569 P1 S0 D3					;Drive 1 Y / Front
M569 P2 S1 D3					;Drive 2 Z Front Left
M569 P3 S1 D3					;Drive 3 Z Rear Left
M569 P4 S1 D3					;Drive 4 Z Right
M569 P5 S1 D3					;Drive 5 Extruder

;Axis configuration
M669 K1							;corexy mode
M584 X0 Y1 Z2:3:4 E5			; Map X to drive 0 Y to drive 1, Z to drives 2, 3, 4, and E to drive 5

;Leadscrew locations
M671 X-13.6:-13.3:340 Y22:278:149 S7.5

M350 X16 Y16 Z16 E16 I1				;set 16x microstepping for axes with interpolation
M906 X2000 Y2000 Z1200 E1200 I10	;Set motor currents (mA)

; Fully commissioned speeds.
M201 X3000 Y8000 Z100 E1500       ; Accelerations (mm/s^2)
M203 X24000 Y24000 Z800 E3600     ; Maximum speeds (mm/min)
M566 X1000 Y1000 Z100 E600 P1     ; Maximum jerk speeds mm/minute

;M201 X6000 Y6000 Z100 E800			;Accelerations (mm/s^2)
;M203 X24000 Y24000 Z900 E7200			;Maximum speeds (mm/min) 
;M566 X1000 Y1000 Z100 E300 P1			;Maximum jerk speeds mm/minute 

M208 X300 Y290 Z300				;set axis maxima and high homing switch positions (adjust to suit your machine)
M208 X0 Y0 Z0 S1				;set axis minima and low homing switch positions (adjust to make X=0 and Y=0 the edges of the bed)
M92 X200 Y201 Z1600 E696	;Orbiter	;steps/mm
;M92 X200 Y201 Z1600 E835    ;Bowden

;Mesh leveling 
M557 X80.0:280.0 Y60.0:280.0 S60

; End Stops
M574 X1 S1 P"io1.in"				;Map the X endstop to io1.in
M574 Y1 S1 P"io2.in"				;May the Y endstop to io2.in

; Filament Sensor 
M591 D0 P5 C"io5.in" R60:200 E3.0 S0 A1 L0.406; Duet3D laser sensor for extruder drive 0, connected to IO 5, tolerance 60 to 200%, 3mm comparison length, disabled
 
; Thermistors
M308 S2 P"temp3" Y"thermistor" A"heatsink_temp" T100000 B3950
M308 S0 P"temp2" Y"thermistor" A"plate_heat" T100000 B3950 H0 L0 			;Plate thermistor - connected to temp2
M308 S1 P"temp1" Y"thermistor" A"e0_heat" T100000 B4725 C7.06e-8 H0 L0		;E3D thermistor - connected to e0_heat

;Define Heaters
M950 H0 C"out0" T0				;Bed heater is on out0
M950 H1 C"out1" T1				;Hotend heater is on out1

;Define Bed
M140 H0

M307 H0 R0.394 K0.158:0.000 D13.55 E1.35 S1.00 B0	;Bed Duet 3 configuration
M307 H1 R3.039 K0.399:0.000 D3.49 E1.35 S1.00 B0 V23.8 ;Phaetus Hotend Duet 3 configuration
;old hotend value M307 H1 A310.1 C144.7 D2.6 S1.00 B0 		
;old bed value M307 H0 A90.0 C700.0 D10.0 S1.00 B0

M570 H0 S360					;Hot end may be a little slow to heat up so allow it 180 seconds
M143 S285						;Set max hotend temperature			

; Fans
M950 F0 C"out6"	Q100				;Layer fan on  "out6" connector
M106 P0 S0 H-1 C"Part Fan"			;Layer fan						
M950 F1 C"!out4+out4.tach" Q250     ;Hotend fan on "out4" connector with PWM
M106 P1 S0 H-1 C"Hotend Fan" 	    ;M106 P1 S255 H1 T50 "Hotend Fan"	;enable thermostatic mode for hotend fan
;M950 F2 C"out4" Q100				;Extruder fan on "out4" connector	
;M106 P2 S0 H-1 C"Extruder Fan"		;Extruder fan

; Accelerometer
M955 P0 C"spi.cs3+spi.cs2" ; all wires connected to temp DB connector

; Input Shaping
M593 P"mzv" F70

; Tool definitions
M563 P0 D0 H1 F0	                        ;Define tool 0 uses extruder 1, heater 1 and fan 1
G10 P0 S0 R0                        		;Set tool 0 operating and standby temperatures

; Pressure Advance
M572 D0 S0.01 ; set extruder 0 pressure advance to 0.01 seconds

; Euclid
M558 P5 C"^io7.in" H5 F200 T20000

; Tip: A larger trigger height in G31 moves you CLOSER to the bed
G31 X0 Y40 Z3.0 P25 			;Nozzle-X set the probe offsets
;G31 X0 Y40 Z2.8 P25 			;0.25mm nozzle set the probe offsets

T0						;select first hot end

I am not sure what is going on and any help would be much appreciated

dc42

@agmeadows if the crashes are truly random then the most likely cause is electrostatic discharge between the hot end metalwork and the hot end temperature sensor, which occurs when extrusion causes static build-up on the hot end. The fix is to ground the hot end metalwork to Duet ground, either directly to the ground VIN terminal, or via a resistor (e.g. 100K) to any Duet ground connection. The resistor is to protect the Duet in case of a short between the hot end heater and the hot end metalwork, which can happen if the heater cartridge develops a fault.

However, the M122 report you posted does not appear to me to be typical of random crashes caused by ESD; so if you get continued random crashes then please post M122 reports taken after each crash so that I can see if there is a pattern.

agmeadows

@dc42 Thank you for the information. ESD sounds like it might be the culprit. I switch my build plate to a PEI sheet yesterday and that may be a contributing factor. I was using Gekotek sheets for a while and never had an issue. I have made sure the nozzle now has continuity to ground and am testing now.

Are you aware of any other ways to insulate from static, specifically the temperature sensor? Having one more wire run in an already full loom is less than ideal. Also, the hotend is coated so getting a good connection is difficult.

agmeadows

@dc42 running a dedicated grounding wire seems to have solved the problem. Do you have suggestions for other methods to ground the hot end or to isolate the sensor? Maybe how to integrate an ESD protection circuit with the sensor?

dc42

@agmeadows where no ground connection is readily available at the hot end, a high value resistor (e.g. 100K) between the hot end metalwork and the VSSA side of the sensor can be used.

agmeadows

@dc42 turns out I spoke too soon... The printer crashed again midway into a print. I made changes to the way I ran the wire and am testing again. Here is the log:

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.5 (2022-11-30 19:35:23) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-956BA-NA3TJ-6JTD8-3SJ6J-1UA2U
Used output buffers: 3 of 40 (22 max)
=== RTOS ===
Static ram: 152760
Dynamic ram: 97584 of which 76 recycled
Never used RAM 100252, free system stack 204 words
Tasks: NETWORK(ready,151.5%,230) ETHERNET(notifyWait,0.3%,401) HEAT(notifyWait,0.1%,322) Move(notifyWait,0.0%,351) CanReceiv(notifyWait,0.0%,944) CanSender(notifyWait,0.0%,336) CanClock(delaying,0.1%,333) TMC(notifyWait,41.6%,91) MAIN(running,325.0%,925) IDLE(ready,0.0%,30), total 518.5%
Owned mutexes:
=== Platform ===
Last reset 01:58:14 ago, cause: software
Last software reset at 2023-04-05 14:10, reason: MemoryProtectionFault mmarValid daccViol, Platform spinning, available RAM 95836, slot 2
Software reset code 0x4160 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x00427804 BFAR 0x61c30f6c SP 0x2041b608 Task MAIN Freestk 1738 ok
Stack: 00000000 00000000 9e3cf497 00000000 0000036c 00476159 00474eae 21000000 00000000 204194f4 20419a84 8100001b 00493838 2042474c 00000000 00000000 00000000 00000000 20430238 00000000 20430b84 00000000 00000189 00476159 20430b3c 00474ebf 07ef5235
Error status: 0x00
Step timer max interval 130
MCU temperature: min 36.6, current 37.3, max 40.0
Supply voltage: min 23.8, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 8, reads 15072, writes 15 timeouts 0
Driver 1: standstill, SG min 0, mspos 568, reads 15073, writes 15 timeouts 0
Driver 2: standstill, SG min 0, mspos 184, reads 15073, writes 15 timeouts 0
Driver 3: standstill, SG min 0, mspos 696, reads 15073, writes 15 timeouts 0
Driver 4: standstill, SG min 0, mspos 616, reads 15073, writes 15 timeouts 0
Driver 5: standstill, SG min 0, mspos 344, reads 15073, writes 15 timeouts 0
Date/time: 2023-04-05 16:08:30
Slowest loop: 4.58ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== Filament sensors ===
Extruder 0: pos -6.54, errs: frame 5 parity 0 ovrun 0 pol 0 ovdue 0
=== CAN ===
Messages queued 63853, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 50), ts 35475/0/0
Tx timeouts 0,0,35474,0,0,28377 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 2.77ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 2 of 8
= Ethernet =
State: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0