Duet 2 Wifi rebooting/crashing during print

bwnance

Hi!

I've had my duet for several months without issue - recently it has been rebooting randomly during prints, roughly anywhere from 10 - 60 mins into the print. The only thing I changed recently was adding a berd-air max kit, which is wired directly onto a fan header.

I'm on FW 3.1.1, but I've tried swapping to the 3.2 beta and the issue persists. Suspecting the SD Card, I did try swapping SD cards and that mitigated the issue for a little bit, but it started happening again not long after.

Here's my M122 report for last night's crash:

=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
Board ID: 08DJM-9178L-L2MSD-6J1FA-3S46N-KSFYP
Used output buffers: 3 of 24 (21 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93236 of which 60 recycled
Exception stack ram used: 312
Never used ram: 9484
Tasks: NETWORK(blocked,384) HEAT(blocked,1224) MAIN(running,1880) IDLE(ready,80)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 07:56:02 ago, cause: software
Last software reset at 2020-11-05 03:02, reason: Hard fault, spinning module PrintMonitor, available RAM 8996 bytes (slot 3)
Software reset code 0x4069 HFSR 0x40000000 CFSR 0x00010000 ICSR 0x0441f803 BFAR 0xe000ed38 SP 0x200053fc Task MAIN
Stack: 0042ea7f 0042b6f2 a5038400 00000000 00000000 00000000 00000000 20005480 00000001 10000000 e000ed04 
Error status: 0
MCU temperature: min 23.3, current 28.5, max 28.8
Supply voltage: min 23.9, current 24.1, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2020-11-05 10:58:06
Cache data hit count 4294967295
Slowest loop: 5.47ms; fastest: 0.14ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 2.0ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 200.90ms; fastest: 0.09ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 1
WiFi firmware version 1.23
WiFi MAC address dc:4f:22:6e:6c:ca
WiFi Vcc 3.43, reset reason Unknown
WiFi flash size 4194304, free heap 18416
WiFi IP address 192.168.86.26
WiFi signal strength -49dBm, reconnections 0, sleep mode modem
Socket states: 0 0 0 0 0 0 0 0

Thanks!

Phaedrux

How is the berd air wired in exactly? I seem to recall seeing that pumps and such might require a fly back diode or some such.

Do the crashes persist with the berd air pump disconnected?

Can you post your config.g as well?

bwnance

@Phaedrux
The berd air is wired without a flyback, directly to the fan0 header - as far as i could tell, a flyback diode isn't required for newer revisions of the duet, as each fan header has a flyback diode built in.

I'll try a print without the berd air wired in and get back to you.

config.g is here:

; Configuration file for Duet WiFi (firmware version 3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.1.4 on Wed Sep 09 2020 23:24:59 GMT-0400 (Eastern Daylight Time)

; General preferences
G90                                                 ; send absolute coordinates...
M83                                                 ; ...but relative extruder moves
M550 P"Casey"                                       ; set printer name
M665 R140.3 L288 B120 H530                          ; Set delta radius, diagonal rod length, printable radius and homed height
M666 X0 Y0 Z0                                       ; put your endstop adjustments here, or let auto calibration find them

; Network
M552 S1                                             ; enable network
M586 P0 S1                                          ; enable HTTP
M586 P1 S0                                          ; disable FTP
M586 P2 S0                                          ; disable Telnet

; Drives
M569 P0 S1                                          ; physical drive 0 goes forwards
M569 P1 S1                                          ; physical drive 1 goes forwards
M569 P2 S1                                          ; physical drive 2 goes forwards
M569 P3 S1                                          ; physical drive 3 goes forwards
M584 X0 Y1 Z2 E3                                    ; set drive mapping
M350 X16 Y16 Z16 E16 I1                             ; configure microstepping with interpolation
M92 X80.00 Y80.00 Z80.00 E397.50                    ; set steps per mm
M566 X1200.00 Y1200.00 Z1200.00 E1200.00            ; set maximum instantaneous speed changes (mm/min)
M203 X18000.00 Y18000.00 Z18000.00 E1200.00         ; set maximum speeds (mm/min)
M201 X1000.00 Y1000.00 Z1000.00 E1000.00            ; set accelerations (mm/s^2)
M906 X1200 Y1200 Z1200 E1000 I30                     ; set motor currents (mA) and motor idle factor in per cent
M84 S30                                             ; Set idle timeout

; Axis Limits
M208 Z0 S1                                          ; set minimum Z

; Endstops
M574 X2 S1 P"xstop"                                 ; configure active-high endstop for high end on X via pin xstop
M574 Y2 S1 P"ystop"                                 ; configure active-high endstop for high end on Y via pin ystop
M574 Z2 S1 P"zstop"                                 ; configure active-high endstop for high end on Z via pin zstop

; Z-Probe
M558 P5 R0.4 C"zprobe.in+zprobe.mod" H5 F1200 T6000 ; set Z probe type to effector and the dive height + speeds
G31 P100 X0 Y0 Z-0.24                               ; set Z probe trigger value, offset and trigger height
M557 R90 S20                                        ; define mesh grid

; Heaters
M308 S0 P"bedtemp" Y"thermistor" T100000 B4138      ; configure sensor 0 as thermistor on pin bedtemp
M950 H0 C"bedheat" T0                               ; create bed heater output on bedheat and map it to sensor 0
M307 H0 B1 S1.00                                    ; enable bang-bang mode for the bed heater and set PWM limit
M140 H0                                             ; map heated bed to heater 0
M143 H0 S120                                        ; set temperature limit for heater 0 to 120C
M308 S1 P"e0temp" Y"thermistor" T100000 B4138       ; configure sensor 1 as thermistor on pin e0temp
M950 H1 C"e0heat" T1                                ; create nozzle heater output on e0heat and map it to sensor 1
M307 H1 B0 S1.00                                    ; disable bang-bang mode for heater  and set PWM limit

;pressure advance
M572 D0 S0.25
; Fans

; PART FAN BerdAir
M950 F0 C"fan0" Q25500                             ; create fan 1 on pin fan0 and set its frequency
M106 P0 H-1 C"Print Fan" F25500                    ; set fan 1 value. Thermostatic control is turned off

; BOARD FAN
M950 F1 C"fan1" Q50                                ; create fan 0 on pin fan1 and set its frequency
M106 P1 S1 H0 T45 C"Board Fan"                     ; set fan 0 value. Thermostatic control is turned on

; HOTEND FAN
M950 F2 C"fan2" Q25000                                ; create fan 2 on pin fan2 and set its frequency
M106 P2 S0.5 H1 T45 C"Hotend Fan"                  ; set fan 2 value. Thermostatic control is turned on

; Tools
M563 P0 D0 H1 F0                                    ; define tool 0
G10 P0 X0 Y0 Z0                                     ; set tool 0 axis offsets
G10 P0 R0 S0                                        ; set initial tool 0 active and standby temperatures to 0C

; Custom settings are not defined

; Miscellaneous
M575 P1 S1 B57600                                   ; enable support for PanelDue
M501

bwnance

I also just remembered that the other change I did around the time the crashes started was swapping to a mosquito hotend.

I should also probably post what my machine is:

Custom Delta
Duet 2 Wifi
duet smart effector
24v 350w meanwell PSU
berd air max
mosquito mounted with the 713maker smart effector adapter
24v heated bed, wired directly into the duet
e3d hemera, mounted as a flying extruder
paneldue 7i

Phaedrux

Neither the bird air nor the osquito seem likely to cause the fault.

Will have to wait to see what @dc42 thinks of the error message.

In the meantime, please continue to gather M122 reports if it crashes.

Perhaps share a gcode file that produces the crash.

bwnance

@Phaedrux
Another crash from this morning (berd air still connected)

The gcode file is too large to attach here -- is there a preferred hosting site for me to upload the file?

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
Board ID: 08DJM-9178L-L2MSD-6J1FA-3S46N-KSFYP
Used output buffers: 3 of 24 (11 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93236 of which 60 recycled
Exception stack ram used: 480
Never used ram: 9316
Tasks: NETWORK(ready,368) HEAT(blocked,1224) MAIN(running,1848) IDLE(ready,80)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:12:58 ago, cause: software
Last software reset at 2020-11-05 11:55, reason: Hard fault, spinning module PrintMonitor, available RAM 8996 bytes (slot 1)
Software reset code 0x4069 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x0441f803 BFAR 0xe000ed38 SP 0x20005414 Task MAIN
Stack: 0042eaf1 20003d50 60030000 00000001 00000009 00000014 0044eaab 0044eaae 21030000 3f000000 41492f1a 
Error status: 0
MCU temperature: min 24.1, current 28.4, max 28.7
Supply voltage: min 23.9, current 24.1, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 0/158
Driver 1: standstill, SG min/max 5/150
Driver 2: standstill, SG min/max 0/166
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2020-11-05 12:08:02
Cache data hit count 1269324432
Slowest loop: 5.60ms; fastest: 0.14ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.2ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 163, MaxWait: 19944ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 5, completed moves: 5, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 15.91ms; fastest: 0.00ms
Responder states: HTTP(2) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address dc:4f:22:6e:6c:ca
WiFi Vcc 3.43, reset reason Unknown
WiFi flash size 4194304, free heap 21840
WiFi IP address 192.168.86.26
WiFi signal strength -53dBm, reconnections 0, sleep mode modem
Socket states: 4 0 0 0 0 0 0 0

Phaedrux

@bwnance said in Duet 2 Wifi rebooting/crashing during print:

is there a preferred hosting site for me to upload the file?

You may be able to zip it and change the file extension to .txt and upload it here if it's within 4Mb (i think) otherwise dropbox or google drive.

bwnance

@Phaedrux
Good call! Attached is the zipped gcode file, renamed to .txt:

vibrationDampener2.gcode.txt

Phaedrux

What's in the macro here:

M98 P"0:/macros/start.g"

bwnance

@Phaedrux
I've got a little bit of a macro tree:
start.g:

G28
M98 P"0:/macros/brushclean.g"
M98 P"0:/macros/bottomhome.g"
G29 S1 ; load height map

brushclean.g:

G0 Z25 F12000
M564 S0 ; disable limits
G0 X0 Y0 Z12 F3000 ; middle of bed, z=5
G0 X0 Y-130 ; middle of brush
G0 Z8 ; drop nozzle slightly
G0 X15 ; 
G0 Y-128 X-15
G0 Y-125 X15
G0 Y-125 X0
G0 Z12
G0 X0 Y0
M564 S1 ; enable limits again

bottomhome.g:

M106 P0 S0
G0 Z25 F12000
G30 
G0 Z15 F12000

bwnance

@Phaedrux @dc42 Hey y'all, just bumping this again -- here's another crash report from today, along with the offending gcode:
tree_new.txt

=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
Board ID: 08DJM-9178L-L2MSD-6J1FA-3S46N-KSFYP
Used output buffers: 3 of 24 (11 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93236 of which 60 recycled
Exception stack ram used: 304
Never used ram: 9492
Tasks: NETWORK(ready,368) HEAT(blocked,1224) MAIN(running,1880) IDLE(ready,80)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:29:46 ago, cause: software
Last software reset at 2020-12-14 23:54, reason: Hard fault, spinning module GCodes, available RAM 8940 bytes (slot 2)
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x04427003 BFAR 0xa5a5a5a5 SP 0x2001ffbc Task MAIN
Stack: 0040f451 0040f452 0100001f 200052f0 200052ef 20008c2c 20003d50 a5a5a5a5 ffffffed 00000000 00f00000 
Error status: 0
MCU temperature: min 23.4, current 28.8, max 29.5
Supply voltage: min 23.9, current 24.1, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2020-12-15 00:24:45
Cache data hit count 2932587391
Slowest loop: 3.44ms; fastest: 0.14ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 0.9ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 15.73ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address dc:4f:22:6e:6c:ca
WiFi Vcc 3.43, reset reason Unknown
WiFi flash size 4194304, free heap 21056
WiFi IP address 192.168.86.34
WiFi signal strength -57dBm, reconnections 0, sleep mode modem
Socket states: 0 0 0 0 0 0 0 0

bwnance

and another one just now, the more the merrier, i guess

=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
Board ID: 08DJM-9178L-L2MSD-6J1FA-3S46N-KSFYP
Used output buffers: 3 of 24 (11 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93236 of which 60 recycled
Exception stack ram used: 472
Never used ram: 9324
Tasks: NETWORK(ready,408) HEAT(blocked,1224) MAIN(running,1840) IDLE(ready,80)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:00:37 ago, cause: software
Last software reset at 2020-12-15 02:33, reason: Hard fault, spinning module none, available RAM 8892 bytes (slot 3)
Software reset code 0x4071 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x0441f003 BFAR 0x91200200 SP 0x2001ff8c Task NETW
Stack: 00454e15 00454e20 6100000e 40090000 0041d011 00000000 20002860 10000000 2000b98c 00000000 200034f0 
Error status: 0
MCU temperature: min 24.2, current 24.8, max 25.0
Supply voltage: min 23.9, current 24.1, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: ok, SG min/max not available
Driver 1: ok, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2020-12-15 02:34:35
Cache data hit count 58687792
Slowest loop: 3.66ms; fastest: 0.14ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 8
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.0ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 166, MinFreeDm: 166, MaxWait: 18624ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 1, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: 3
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by HTTP
HTTP is idle in state(s) 1 0 8, running macro
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 15.75ms; fastest: 0.00ms
Responder states: HTTP(2) HTTP(4) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address dc:4f:22:6e:6c:ca
WiFi Vcc 3.43, reset reason Unknown
WiFi flash size 4194304, free heap 21408
WiFi IP address 192.168.86.34
WiFi signal strength -55dBm, reconnections 0, sleep mode modem
Socket states: 4 2 0 0 0 0 0 0

markz

Since no one has jumped in here, my usual opinion when I see something like this is that (assuming the board is ok) it's either a power supply issue or a voltage noise issue.

How much power does your heated bed use? Is it enough to impact the power supply?

How is the 24V to the pump connected? Could noise on the 24V line be caused by the pump pwm?

bwnance

@markz
Resistance on my heated bed (measured at connector on board) is 3.3ohms, it's a 24v bed, so that would be a ~175watt bed, correct? I'm using a Mean Well 24v 350w PSU and my hotend is 50w, so i should have plenty of extra room, wattage wise, right?

24v to the pump is over a fan header on the board (AFAIK newer duets have the flyback diode on all fan inputs internally), I'm not sure how to diagnose if i have noise on the 24v line.

markz

@bwnance thanks.

The pump has two inputs (kind of) power and control. I know the control line goes to the fan pins but where do you pull off the 24V supply for the pump - is that from the power supply or the duet board somewhere? Are you getting the 24v from the fan header?

bwnance

@markz I'm not using "EZ-Switch" board that was packaged with the berd-air, so both lines are hooked up to the fan header -- as best as i could tell before installing, this was fine, since the fan headers have flybacks wired into them

Phaedrux

Can you try 3.2 Beta 4? It's believed that a fix for this crash may already be present there.

https://github.com/Duet3D/RepRapFirmware/releases

bwnance

@Phaedrux just upgraded, will do a few test prints! We shall see...

Phaedrux

@bwnance Just realized there is an odd ordering problem on that page that doesn't list the most recent at the top.

https://github.com/Duet3D/RepRapFirmware/releases/tag/3.2beta4.1

Beta 4.1 can be found here.

bwnance

@Phaedrux ah, ty! gonna let this print finish on beta 4, then will update to 4.1