Mid print stalls on new Duet3 printer

Phaedrux

@Olby said in Mid print stalls on new Duet3 printer:

I hooked up a SBC with the hope that would solve it (and a different issue more about that later) but still run into it.

Are you saying it was happening in standalone mode and you've since switched to SBC mode and it's still happening?

Can you share your config.g as well?

Olby

Yes both issues are/were consistent both with SBC and stand alone. The stall happens about once every 10-20h of printing so far but the machine is fairly new so not much data points.

Here is my config.g:

	; Jubilee CoreXY ToolChanging Printer - Config File
	; This file intended for Duet 3 hardware, main board plus two expansion boards
	; As of 4/08/2020 INCOMPLETE AND ONLY PARTIALLY TESTED.  See the discord for more info. 

	; Name and Identification
	; Printer name is now set on the R Pi. 


; Communication and general -------------------------------------------------------------------------------------------------------------------------------------
    ; M550 P"Bet"                                 ; set printer name

	; Networking
	M552 P192.168.2.167 S1						; Use Ethernet with a static IP
	M553 P255.255.255.0		                    ; Netmask
	M554 192.168.1.1							; Gateway
	M586 P1 S1									; Enable FTP
	M586 P2 S1									; Enable Telnet

	; Debugging
	M111 S0										; Debug off
	M929 P"eventlog.txt" S1						; Start logging to file eventlog.txt

	; General Preferences
	M555 P2										; Set Marlin-style output
	G21											; Set dimensions to millimetres
	G90											; Send absolute coordinates...
	M83											; ...but relative extruder moves



; Axis to driver mapping -----------------------------------------------------------------------------------------------------------------------------------------
	M584 X5 Y4 U3 Z1:2:0						; X and Y for CoreXY.  U for toolchanger lock. Z has three drivers for kinematic bed suspension. 
	;M584 E1.0:1.1:2.0:2.1						; Extruders for four tools. 
    M584 E1.0:1.1								; Extruders for two tools. 

	M569 P5 S0									; Drive 0 | X stepper
	M569 P4 S0									; Drive 1 | Y Stepper
	M569 P3 S0									; Drive 2 | U Tool Changer Lock
	M569 P0 S0									; Drive 5 | Back Z
	M569 P1 S0									; Drive 3 | Front Left Z
	M569 P2 S0									; Drive 4 | Front Right Z

	
	; End of main board drivers.  Expansion boards have three each.
	; Expansion 0
	M569 P1.0 S0									; Drive 6 | Extruder T0
	M569 P1.1 S0									; Drive 7 | Extruder T1
	
	; Expansion 1
	;M569 P5 S0									; Drive 8 | Extruder T2
	;M569 P9 S0									; Drive 9 | Extruder T3



; Kinematics -----------------------------------------------------------------------------------------------------------------------------------------------------
	M669 K1										; CoreXY mode

	; Kinematic bed ball locations.
	; Locations are extracted from CAD model assuming lower left build plate corner is (0, 0) on a 305x305mm plate.
	M671 X297:2.5:150 Y313.5:313.5:-16.5 S10 ; Front Left: (297.5, 313.5) | Front Right: (2.5, 313.5) | Back: (150, -16.5)



; Axis and motor configuration -----------------------------------------------------------------------------------------------------------------------------------
	M350 X16 Y16 Z16 E16 U4 I1              ; Set 16x microstepping for xyz axes & extruder, 4x for toolchanger lock. Use interpolation.
	
    M906 X2263 Y2263                        ; LDO XY motor currents (mA). 2263mA is 80% of 2828mA Peak Current. (2828mA Peak is 2000mA RMS)
                                            ; Do not exceed 90% of full XY motor current rating without heatsinking the XY motor drivers.
    M906 Z1600                              ; LDO ZZZ Motor currents (mA). Conservative.

	M906 E1250                              ; Motor currents E (mA)

    M906 U670 I60                           ; LDO Toolchanger Elastic Lock motor current (mA) and idle motor percentage.

	M201 X1000 Y1000                        ; XY Acceleration. Conservative. 1500+ possible.
    M201 Z100                               ; LDO ZZZ Acceleration
    M201 E1300                              ; E Acceleration
    M201 U800                               ; LDO U Accelerations (mm/s^2) conservative
	M203 X13000 Y13000 Z800 E8000 U9000		    ; Maximum speeds (mm/min)
	M566 X480 Y480 Z500 E3000 U50     		; Maximum jerk speeds mm/minute

	M92 X200 Y200								; Steps/mm for X,Y with 16 tooth pulleys (preferred). 
	M92 Z3200									; Steps/mm for Z - TR8*2 / 0.9 deg stepper
	M92 U30.578 								; Steps/mm for tool lock geared motor. 
	M92 E830									; Extruder - 0.9 deg/step



; Endstops and probes --------------------------------------------------------------------------------------------------------------------------------------------
	M574 U1 S1 P"io0.in"						; Set homing switch configuration U1 = low-end, S1 = active-high (NC)
	M574 X1 S1 P"io1.in"						; Set homing switch configuration X1 = low-end, S1 = active-high (NC)
	M574 Y1 S1 P"io2.in"						; Set homing switch configuration Y1 = low-end, S1 = active-high (NC)

	M558 P5 C"io3.in" H3 A1 T6000 S0.02
	;M558 K0 P5 C"io4.in" H5 A5 T6000 S0.02		; Z probe - Set the height of the bed when homing G28.  Combined with content of bed.g as invoked by G32, levels bed. Also used for Mesh. 
												; P5 = Switch, NC
												; C  = Input Connector
												; Hn = dive height
												;   A bigger dive height prevents a situation where the bed is out of alignment by more than the dive height
												;   on any corner, which can crash the hot-end into the bed while moving the head in XY.
												;   Probing speed and travel speed are similarly reduced in case the Z probe isn't connected properly (or
												;   disconnects later after moving to a point) giving the user more time to stop.
												; An = Number of times to probe each point. 
												; Tnnn = Travel speed between probe points. 
												; Snnn = Tolerance when probing multiple times. Two readings inside this window and we move on. 
	;M558 K1 P5 C"!io4.in"                   	; create probe #1 for use with M585, active low


	; Set axis software limits and min/max switch-triggering positions.
	; Adjusted such that (0,0) lies at the lower left corner of a 300x300mm square in the 305mmx305mm build plate.
	M208 X-13.75:313.75 Y-44:341 Z-0.2:295
	M208 U0:200									; Set Elastic Lock (U axis) max rotation angle



; Thermistors and heaters ---------------------------------------------------------------------------------------------------------------------------------------
	; ***Temperature sensors***
	; Main Board
	M308 S0 P"0.temp0" Y"thermistor" T100000 B3950 A"Bed"		; Silicone Pad
	;M308 S5 P"0.temp1" Y"thermistor" T10850 B2900 A"BedT"		; Theramal Lug these are not factory but calibrated so that the lug displays bed temp
																; based on measurements with IR from 19.5c-60c (currently only monitored)
    ;M308 S0 P"spi.cs1" Y"rtd-max31865" A"spi.cs1" ;
	;M308 S1 P"spi.cs1" Y"rtd-max31865" a"spi.cs1" ;

	; Expansion 0
	M308 S1 P"spi.cs1" Y"rtd-max31865" A"HE1"							; Configure extruder 1 temperature sensor - PT100 sensor
	; M308 S2 P"1.temp1" Y"thermistor" T100000 B3950 A"HE2"
	
	; Expansion 1
	;M308 S3 P"2.temp0" Y"thermistor" T100000 B3950 A"HE3"
	;M308 S4 P"2.temp1" Y"thermistor" T100000 B3950 A"HE4"


	; ***Heaters***
	; Main Board
	M950 H0 C"0.out0" T0						; Bed.  H = Heater 0, C is output for heater itself, T = Temperature sensor (defined above)
	
	; Expansion 0
	M950 H1 C"0.out3" T1						; Heater for extruder out tool 0
    ; M307 H1 A1252.3 C361.3 D5.3 V24.0 B0        ; Reported from pid tuning without sock
    M307 H1 A811.4 C309.4 D4.7 V24.0 B0         ; With sock
	;M950 H2 C"1.out1" T2						; Heater for extruder out tool 1
	
	; Expansion 1
	;M950 H3 C"2.out0" T3						; Heater for extruder out tool 2
	;M950 H4 C"2.out1" T4						; Heater for extruder out tool 3


	;*** Max Temp ***
	; Main Board
    M570 S30                                    ; Print will be terminated if a heater fault is not reset within 30 minutes.
	M143 H0 S130								; Set maximum temperature for bed to 130C
	
	; Expansion 0
	M143 H1 S300								; Set maximum temperature for hotend to 300C
	; M143 H2 S300								; Set maximum temperature for hotend to 300C
	
	; Expansion 1
	;M143 H3 S300								; Set maximum temperature for hotend to 300C
	;M143 H4 S300								; Set maximum temperature for hotend to 300C	
	
	
	; ***Further setup for bed***
	M140 H0										; map heated bed to heater 0

    M307 H0 A589.8 C589.8 D2.2 V24.1 B0        ; Keenovo 750w 220v built in thermistor mandala rose bed



; Fans -------------------------------------------------------------------------------------------------------------------------------------------------------
	; ***Fans for hot end cooling***
	; Expansion 0
	M950 F1 C"0.out8"
	M106 P1 S255 H1 T45							; S = Speed of fan Px, Hxx = heater for thermo mode, T = temps for thermo mode.
	;M950 F2 C"1.out7"
	;M106 P2 S255 H2 T45							; S = Speed of fan Px, Hxx = heater for thermo mode, T = temps for thermo mode.
	
	; Expansion 1
	;M950 F3 C"2.out6"
	;M106 P3 S255 H3 T45						; S = Speed of fan Px, Hxx = heater for thermo mode, T = temps for thermo mode.
	;M950 F4 C"2.out7"
	;M106 P4 S255 H4 T45						; S = Speed of fan Px, Hxx = heater for thermo mode, T = temps for thermo mode.


	; ***Fans for print cooling***
	; Expansion 0
	M950 F5 C"0.out7" 
	M106 P5 C"PrintCool5"
	;M950 F6 C"1.out4" 
	;M106 P6 C"PrintCool6"
	
	; Expansion 0
	;M950 F7 C"2.out3" 
	;M106 P7 C"PrintCool7"
	;M950 F8 C"2.out4" 
	;M106 P8 C"PrintCool8"



; Tool definitions --------------------------------------------------------------------------------------------------------------------------------------------
	; Expansion 0
	M563 P0 S"Tool 0" D0 H1 F5					; Px = Tool number, Dx = Drive Number (start at 0, after movement drives), H1 = Heater Number, Fx = Fan number print cooling fan
	G10  P0 S0 R0								; Set tool 0 operating and standby temperatures(-273 = "off")
	M572 D0 S0.085								; Set pressure advance

	;M563 P1 S"Tool 1" D1 H2 F6					; Px = Tool number, Dx = Drive Number (start at 0, after movement drives), H1 = Heater Number, Fx = Fan number print cooling fan
	;G10  P1 S0 R0								; Set tool 0 operating and standby temperatures(-273 = "off")
	;M572 D1 S0.1								; Set pressure advance on 

	; Expansion 0
	;M563 P2 S"Tool 2" D2 H3 F7					; Px = Tool number, Dx = Drive Number (start at 0, after movement drives), H1 = Heater Number, Fx = Fan number print cooling fan
	;G10  P2 S0 R0								; Set tool 0 operating and standby temperatures(-273 = "off")
	;M572 D2 S0.1								; Set pressure advance on

	;M563 P3 S"Tool 3" D3 H4 F8					; Px = Tool number, Dx = Drive Number (start at 0, after movement drives), H1 = Heater Number, Fx = Fan number print cooling fan
	;G10  P3 S0 R0								; Set tool 0 operating and standby temperatures(-273 = "off")
	;M572 D3 S0.1								; Set pressure advance on 

	M98  P"/sys/Toffsets.g"						; Set tool offsets from the bed. In separate file so test macro can invoke. 



	;M98 P"config-user.g"						; Load custom user config if one exists.

	;M501										; Load saved parameters from non-volatile memory

	;G29 S1

	; Note: you will need to tune the bed heater, and both extruder cartridges before printing.
	; See the following link for more details. https://duet3d.dozuki.com/Wiki/Tuning_the_heater_temperature_control

Olby

Switch to 3.2-beta3 and increase event logging M929 to debug level. Will start a longer print tomorrow to see if i can reproduce the issue. Anything else I can do to get more data to help pinpoint the issue?

Phaedrux

@Olby said in Mid print stalls on new Duet3 printer:

Anything else I can do to get more data to help pinpoint the issue?

If you're still using the SBC you can get some additional info captured with this: https://duet3d.dozuki.com/Wiki/Getting_Started_With_Duet_3#Section_Monitoring_optional

Otherwise capturing M122 afterwards is helpful to see the error codes.

Olby

Turns out that 3.2-beta3 will make this a lot easier to track down the skipped steps that I thought i had worked around. Now I can easily get it to skip steps with just a couple of quick moves.

Same printer, same gcode and 3.1 no missed steps for 9h+. 3.2-beta3 missed steps first layer, missed step after 5 rapid moves and also manage to break the z switch after what i think are missed steps on z. The bed was skewed as if one of my 3 z motors skipped.

I will try to reproduce it on individual axis when i get time after work. Thinking rapid diagonals to test each X/Y individually and rapid z with a dial gauge near each z-motor.

TypQxQ

Try turning off all bed compensations with M561. I am having similar problems with it enabled.

Olby

@TypQxQ Thanks, disabling the bed compensation solved my X/Y skipping issue. Strange that things got worse after the update to 32-beta3, I use the same settings and same bed mesh.

So one down, will start a long print tomorrow to see if I can reproduce the stall with the new firmware.

The version update also brought a new issue. Something has changed since 3.1 that breaks my Z-homing from time to time.

	; Home Z Axis
	M561 ; Disable any Mesh Bed Compensation
	G90 G1 X150 Y150 F10000 ; Move to the center of the bed
	M558 F500 ; Set the probing speed
	G30
	M558 F50 ; Set a slower probing speed
	G30
	G32      ; Run 3-point bed calibration defined in bed.g
	G29 S1   ; Enable Mesh Bed Compensation
	G1 Z5
	G1 X50 Y50 F10000

It is as if the center probe ends with the bed too high and drags the z limit switch against the be so i snaps. Not always but i'm on my second broken switch today after max 10 homings. Never got that behaviour in the old version. (And this is before the bed mesh is applied)

TypQxQ

@Olby Try setting all parameters of M558 each time. I have experienced that it sometimes looses parameters when only setting one of them. Try like this:

M558 P8 C"0.io3.in" F500 H5 T24000 A2 S0.02 ; Set the probing speed
G30
M558 P8 C"0.io3.in" F50 H5 T24000 A2 S0.02 ; Set a slower probing speed

You can also put "G31 Z-1", if you haven't already, in the start to add an extra mm to the switch. You will have to add 1mm to your Z tool offsets until you recalibrate them.

Olby

Updated to 3.2-beta3+3-ch and the skipping issues are gone all works with bed mesh compensation activated.

Will run some long prints this weekend to see if i can reproduce the mid print stop with 100% complete status.

Olby

@TypQxQ Will test that as soon as I soldered on a new switch, thanks.

Olby

Managed to reproduce the mid print halt again. This time with 3.2-beta3.2. Same behaviour ui reports 100% complete but there are lots of gcode in the buffer.

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2-beta3.2 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode)
Board ID: 08DJM-956L2-G43S4-6J9DD-3SJ6S-TB66H
Used output buffers: 1 of 40 (14 max)
=== RTOS ===
Static ram: 122236
Dynamic ram: 139504 of which 68 recycled
Never used RAM 130384, free system stack 122 words
Tasks: Linux(ready,79) HEAT(blocked,296) CanReceiv(blocked,869) CanSender(blocked,335) CanClock(blocked,352) TMC(blocked,20) MAIN(running,669) IDLE(ready,19)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 07:13:50 ago, cause: power up
Last software reset at 2020-11-21 17:24, reason: User, GCodes spinning, available RAM 130384, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0xffffffff Task Linu
Error status: 0x00
MCU temperature: min 8.1, current 26.4, max 26.5
Supply voltage: min 23.8, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.2, current 12.3, max 12.4, under voltage events: 0
Driver 0: position 25039, standstill, reads 22878, writes 30 timeouts 0, SG min/max 0/1023
Driver 1: position -526, standstill, reads 22878, writes 30 timeouts 0, SG min/max 0/1023
Driver 2: position 391749, standstill, reads 22883, writes 26 timeouts 0, SG min/max 0/73
Driver 3: position 0, standstill, reads 22879, writes 31 timeouts 0, SG min/max 0/1023
Driver 4: position 0, standstill, reads 22880, writes 31 timeouts 0, SG min/max 0/1023
Driver 5: position 0, standstill, reads 22881, writes 30 timeouts 0, SG min/max 0/1023
Date/time: 2020-11-26 04:21:46
Slowest loop: 261.09ms; fastest: 0.08ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 375, MinFreeDm: 332, MaxWait: 4966114ms
Bed compensation in use: mesh, comp offset 0.124
=== MainDDARing ===
Scheduled moves 152108, completed moves 152108, StepErrors 0, LaErrors 0, Underruns [0, 0, 3], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages sent 589618, send timeouts 589618, longest wait 1ms for type 6018, free CAN buffers 47
=== SBC interface ===
State: 0, failed transfers: 0
Last transfer: 19ms ago
RX/TX seq numbers: 58211/58212
SPI underruns 0, overruns 0
Number of disconnects: 1, IAP RAM available 0x20a08
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.2.0-beta3
File:
Buffered code: G1 X69.704 Y98.334 E0.02468
Buffered code: G1 X69.880 Y98.698 E0.01369
Buffered code: G1 X70.865 Y99.790 E0.04976
Buffered code: G1 X71.138 Y100.518 E0.02632
Buffered code: G1 X71.094 Y101.246 E0.02468
Buffered code: G1 X70.918 Y101.610 E0.01369
Buffered code: G1 X69.933 Y102.702 E0.04976
Buffered code: G1 X69.661 Y103.430 E0.02632
Buffered code: G1 X69.704 Y104.158 E0.02468
Buffered code: G1 X69.880 Y104.522 E0.01369
Buffered code: G1 X70.865 Y105.614 E0.04976
Buffered code: G1 X71.138 Y106.342 E0.02632
Buffered code: G1 X71.094 Y107.070 E0.02468
Buffered code: G1 X70.918 Y107.434 E0.01369
Buffered code: G1 X69.933 Y108.525 E0.04976
Buffered code: G1 X69.661 Y109.253 E0.02632
Buffered code: G1 X69.704 Y109.981 E0.02468
Buffered code: G1 X69.880 Y110.345 E0.01369
Buffered code: G1 X70.073 Y110.567 E0.00995
Buffered code: G1 X67.558 Y110.567 E0.08513
Buffered code: G1 X67.022 Y109.981 E0.02689
Buffered code: G1 X66.749 Y109.253 E0.02632
Buffered code: G1 X66.792 Y108.525 E0.02468
Buffered code: G1 X66.968 Y108.162 E0.01369
Buffered code: G1 X67.953 Y107.070 E0.04976
Buffered code: G1 X68.226 Y106.342 E0.02632
Buffered code: G1 X68.182 Y105.614 E0.02468
==> 1292 bytes
Trigger:
Finishing macro config.g, started by system
> Next stack level
Pending code: M905 P"2020-11-26" S"06:39:00"
Pending code: M905 P"2020-11-26" S"13:51:18"
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 14.03
File /opt/dsf/sd/gcodes/Part_5.gcode is selected, processing

M122 B1
Diagnostics for board 1:
Duet EXP3HC firmware version 3.2beta3.1 (2020-11-08)
Bootloader ID: not available
Never used RAM 154920, free system stack 154 words
HEAT 40 CanAsync 93 CanRecv 76 TMC 30 MAIN 334 AIN 257
Last reset 07:17:20 ago, cause: power up
Last software reset data not available
Driver 0: position 21928487, 830.0 steps/mm,  standstill, reads 31013, writes 23 timeouts 0, SG min/max 0/661
Driver 1: position 0, 830.0 steps/mm,  standstill, reads 31021, writes 18 timeouts 0, SG min/max 0/0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 31031, writes 11 timeouts 0, SG min/max 0/0
Moves scheduled 485558, completed 485558, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 24.1V, V12: 12.2V
MCU temperature: min 39.0C, current 39.2C, max 39.4C
Ticks since heat task active 175, ADC conversions started 26240916, completed 26240916, timed out 0
Last sensors broadcast 0x00000000 found 0 178 ticks ago, loop time 0
Free CAN buffers: 36, messages lost 721666, duplicates 0, oos 0, busOff 0

The latest beta and a replaced crimp has solved all other issues. But this remains...

The event log is empty and i didn't run with the extra monitoring on the sbc this time.

dc42

There is a known issue with occasional print stalls when running with attached SBC. This was confirmed and fixed yesterday. The fix is in 3.2beta4, which is being tested right now.