Printer hanging during tool change with v3.5.1

Gamefanatic3D

I upgraded to v3.5.1 from v3.4.5. When I attempt a test print utilizing multiple tools and find that at some point during the print when changing tools it hangs. At first, I thought it was due to waiting on a heater, but I could not reproduce the issue when manually changing tools. It only occurs when attempting to print.

When I revert to v3.4.5 I have no issues with the same prints.

Here is an example of one of my tool-changing scripts. I have been able to reproduce the issue with T0 and T2 (did not test against T1).

SYS Files

tfree0.g

; tfree0.g

; Runs at the start of a toolchange if the current tool is tool-0.
; Note: tool offsets are applied at this point unless we preempt commands with G53!
; called when tool 0 is freed
;
; Move Z out of way.
G91              ; relative positioning
G1 Z5 F6000 H2   ; lift Z relative to current position
G90              ; absolute positioning

M98 P"/macros/Tools/T0/Drop" ; Drop Tool

G91              ; relative positioning
G1 Z-5 F6000 H2  ; lift Z relative to current position
G90              ; absolute positioning

;fan off
M106 P1 S0

; Disable Filament Monitor
M591 D0 P3 c{global.tool0_mfm_PIN} S0	; Duet3D rotating magnet sensor disabled


; Clear Tool Specific variables
set global.currentTool_fan = ""

tpost0.g

; tpost0.g

; called after tool 0 has been selected
;
M703		; Load filament configuration

; Squeeze Filament out.
M116 P0 S3
M98 P"/sys/prime.g"

; Pickup Tool
M98 P"/macros/Tools/T0/Pickup"

; restore print cooling fan speed
;M106 P1 R2
M106 P1 S{state.restorePoints[2].fanPwm}

if global.restoreToolPosition = 1
	G0 R2 X0 Y0 F7000 ; Restore to X and Y position prior to tool change
	
G1 R2 Z0                     ; Restore prior Z position before tool change was initiated.
                             ; Note: tool tip position is automatically saved to slot 2 upon the start of a tool change.
                             ; Restore Z first so we don't crash the tool on retraction.
G1 E{tools[{state.currentTool}].retraction.length} ; Prime Nozzle.

tpre0.g

; tpre0.g

; called before tool 0 is selected
;
; Check if we have another tool selected
; Is this even necessary with our tool detection script in play?

M98 P"/sys/thome_check.g"

G60 S0 ; Save Current Position to Slot 0


;;;;;;;;;;;;;;;;;;;;;
;; FILAMENT SENSOR ;;
;;;;;;;;;;;;;;;;;;;;;
; Duet3D rotating magnet sensor extruder drive 0 is connected to E0, enabled, sensitivity 24.8mm.rev, 70% to 130% tolerance, 3mm detection length 'agc' of 50 to 105
if {global.tool0_mfm_defaults} == false
  M591 D0 P3 c{global.tool0_mfm_PIN} S{global.tool0_mfm_S} R{global.tool0_mfm_R_min}:{global.tool0_mfm_R_max} L{global.tool0_mfm_L} E{global.tool0_mfm_E}
else
  M591 D0 P3 c{global.tool0_mfm_PIN} S1 R20:180 L25.0 E3.0	

;;;;;;;;;;;;;;;;;;;;;;
;;  Tool Variables  ;;
;;;;;;;;;;;;;;;;;;;;;;
set global.currentTool_fan = 1

;;;;;;;;;;;;;;;;;;;;;;
;;    Load Tool     ;;
;;;;;;;;;;;;;;;;;;;;;;
; Turn on heater
if tools[0].active[0] > 0
  M568 P0 A2
elif tools[0].standby[0] > 0
  M568 P0 A1

; Wait for set temperatures to be reached
M116 P0 S2

; Note that commands preempted with G53 will NOT apply the tool offset.
if move.axes[2].machinePosition < abs(tools[0].offsets[2])
  G1 Z{(abs(tools[0].offsets[2])+5)}

; Load T0 Configuration
M98 P"/sys/tconfig0.g"

tconfig0.g

; tconfig0.g

; Retraction
M207 P0 S0.50

;; OFFSET
;; X ;;
; Set tool 0 axis offsets for X coordinate
G10 P0 X-6.71

;; Y ;;
; Set tool 0 axis offsets for Y coordinate
G10 P0 Y35.17

;; Z ;;
; Set tool 0 axis offsets for Z coordinate
G10 P0 Z-1.55

prime.g

; prime.g

if {global.primeTool}!=0
  G92 E0
  G1 F200
  G1 E10 F100
  M106 P{global.currentTool_fan} S255
  G4 S1
  G1 E20 F500
  M106 P{global.currentTool_fan} S0
  G92 E0
  G1 F200
  G1 E{-1*(tools[{state.currentTool}].retraction.length)} F100
  G4 S3

MACROS

Pickup

; Pickup

; This will grab the tool from the dock
;Open Coupler
M98 P"/macros/Tools/Tool-Unlock"

; Pickup Tool
M98 P"/macros/Tools/T0/Dock"

; Lock Coupler
M98 P"/macros/Tools/Tool-Lock"

;; Tool Wipe ;;
; Lock Coupler
M98 P"/macros/Tools/Tool-Wipe-Dock"

; Move away from dock
G1 Y320 F6000

; Identify tool has been picked up.
; We need to know if a tool is in hand.  This
; will execute code to drop tool enabling
; the next step of picking up a tool or homing Z
M28 "/sys/tactive.g"
; Play Alert Sound
M300 S500 P200
G4 P250
M300 S500 P200
G4 P250
M300 S500 P200
; Prompt we detected a tool and will dock (10 second default)
M291 S1 R"Tool Warning" P"Tool 0 detected in carriage.  Preparing to Dock..."
M98 P"/macros/Tools/T0/Drop"
M29

Dock

; Dock

; This file will align the tool head with the tool and

; Tool changer we want to home Y first to pull away from tools.
if !move.axes[0].homed
  ; Home Y
  G28 X
if !move.axes[1].homed
  ; Home X
  G28 Y

; move to prep for lock.
; move to Y 1st for safety
; Prefix commands with G53 to ignore offsets!!
; align with tool
G53 G1 Y300 F8000
G53 G1 Y300 X417


; move to tool
G53 G1 Y375 F4000
G53 G1 Y397 F2500

Drop

; Drop

; This will put the tool back in the dock.
; Dock Tool
M98 P"/macros/Tools/T0/Dock"

; Unlock Coupler
M98 P"/macros/Tools/Tool-Unlock"

; Move away from dock fast in hopes it shakes a sticky tool off like a string.
G53 G1 Y320.5 F12000

; Cleanup tool register
M30 "/sys/tactive.g"

Tool-Lock

; Tool-Lock

M400
M906 C600    ; Raise motor current a bit.
M400
G1 C206 F2500
M400
M906 C400    ; Put it back to normal
M400

Tool-UnLock

; Tool-UnLock

M400
M906 C600    ; Raise motor current a bit.
M400
if !move.axes[3].homed && move.axes[3].userPosition==0
	; We are assuming a tool is loaded and we want to unload it.
	; 
	echo "C-Axis not homed"
	G1 C-83 F2500 H2
else
	G1 C125 F2500
M400
M906 C400    ; Put it back to normal
M400

I believe the problem to be occurring during the tfree0.g as the tool has the active temperature set and I see the bed lower. I would expect it to attempt to pick up the tool at this point, so I'm expecting the hang is occuring between the first G90 and M98 Macro (which I do not see execute).

; Move Z out of way.
G91              ; relative positioning
G1 Z5 F6000 H2   ; lift Z relative to current position
G90              ; absolute positioning

M98 P"/macros/Tools/T0/Drop" ; Drop Tool

I have tried an M108 in hopes it was stuck in a temperature wait loop, but that does not stop it. The only way I have been able to make things work again at this point is to stop the machine through the emergency stop on the Duet PanelDue or Web Console.

No issues with a single tool print or manually cycling through the tools. It's only happening while printing and almost always after cycling through the tools at least once. It will occur in a print job in the same place each time, so it is reproducible.

I have tried to upload the firmware a second time, with the same results.

Moving back to v3.4.5 doing the same print jobs it works without a problem and no changes are required to any macros. Which at least gives me a solution to print, but was hoping to see if I was missing something.

dc42

@Gamefanatic3D from your M122 report I can see that the job is held up waiting for the tpre file to complete, but it hasn't completed because there are three G10 P0 commands pending. These appear to be coming from your tconfig0.g file.

I don't yet know what is causing these commands to remain pending; however if you insert a M400 or G4 P0 command before the M98 P"/sys/tconfig0.g" line at the end of tpre0.g, that may work around the problem.

gloomyandy

@Gamefanatic3D I've not seen any problems with toolchanging on my e3d toolchanger and 3.5.1, so it is not a really generic problem. Things that may help us work out what is going on...

What board are you using?
Can you edit your first post and make sure that it is obvious which script is which, in particular I couldn't work out which one was "active.g" nore could I find "drop".
If possible can you run M122 when the printer is hanging and post the output from that.

Gamefanatic3D

@gloomyandy

Using the Duet2 Wifi + DUEX5. Hopefully, things are updated a bit better. The forum code interpreter seems to be having issues with some of the code I pasted and not giving it proper formatting.

The "tactive.g" file is generated by the "pickup.g" when a tool has been successfully picked up and deleted when it has been successfully returned. I use it in lieu of a switch on the tool to detect if a tool is loaded.

I'll upload an M122 once I get a chance to swap back to the 3.5.1 firmware and run through the paces again.

Gamefanatic3D

Updated to v3.5.1 again.

I ran the attached sliced file and it makes it all the way through to the second T0 (line 1637). Heaters all reach their nominal temperatures, even though my Macro's M116 is only looking at the T0 before it wants to pick it up. The bed will lower, but will not go further.

10x10x10_Cube-T0T1T2-S3D.gcode

I cannot pause the print.
M108 does nothing
M25 - From Web just hangs waiting for a response from the printer.
From the Web Control:
- I can navigate around and send some commands IE) M122
- I can upload a Macro and edit files
- I can click the "Emergency Stop"
From the PanelDue
- I cannot pause the print
- I can move around / navigate
- I can execute Macro's
- I can change the temperatures of all the tools / bed.
- I can click the stop button.

5/5/2024, 2:58:24 PM	M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.5.1 (2024-04-19 14:40:46) running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-9T6BU-FG3S8-6J1F8-3SJ6S-1BL7H
Used output buffers: 3 of 26 (26 max)
=== RTOS ===
Static ram: 23256
Dynamic ram: 78496 of which 28 recycled
Never used RAM 7932, free system stack 108 words
Tasks: NETWORK(1,ready,12.6%,202) HEAT(3,nWait 5,0.2%,309) Move(4,nWait 5,0.4%,258) DUEX(5,nWait 5,0.0%,24) MAIN(1,running,86.7%,747) IDLE(0,ready,0.1%,29), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:17:34 ago, cause: software
Last software reset at 2024-05-05 14:40, reason: User, Gcodes spinning, available RAM 8748, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU temperature: min 41.8, current 46.1, max 46.5
Supply voltage: min 23.8, current 24.2, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/51, heap memory allocated/used/recyclable 2048/1348/140, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 167
Driver 1: standstill, SG min 119
Driver 2: standstill, SG min 0
Driver 3: standstill, SG min 0
Driver 4: standstill, SG min 0
Driver 5: standstill, SG min 0
Driver 6: standstill, SG min n/a
Driver 7: standstill, SG min 0
Driver 8: standstill, SG min 0
Driver 9: standstill, SG min n/a
Driver 10: 
Driver 11: 
Date/time: 2024-05-05 14:58:23
Cache data hit count 4294967295
Slowest loop: 241.50ms; fastest: 0.16ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 2.0ms, write time 5.2ms, max retries 0
=== Move ===
DMs created 83, segments created 24, maxWait 497256ms, bed compensation in use: mesh, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 1.00
no step interrupt scheduled
Moves shaped first try 150, on retry 35, too short 65, wrong shape 408, maybepossible 0
=== DDARing 0 ===
Scheduled moves 735, completed 735, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.3
Heater 1 is on, I-accum = 0.5
=== GCodes ===
Movement locks held by File
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 8
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is doing "G10 P0 X-6.71" in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Q0 segments left 0
Queue 0 has 'G10 P0 Y35.17' for move 735
Queue 0 has 'G10 P0 Z-1.55' for move 735
=== Filament sensors ===
check 7281348 clear 2324874
Extruder 0: pos 2469.73, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
Extruder 1: pos 13973.55, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
Extruder 2: pos 2390.27, errs: frame 0 parity 0 ovrun 0 pol 14549 ovdue 0
=== DueX ===
Read count 1, 0.06 reads/min
=== Network ===
Slowest loop: 131.31ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1.0
MAC address cc:50:e3:14:e3:17
Module reset reason: Turned on by main processor, Vcc 3.32, flash size 4194304, free heap 39504
WiFi IP address 192.168.0.90
Signal strength -48dBm, channel 2, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

Gamefanatic3D

I tested a bit further tonight on this. I ran a Debug log file and added a "M118 P3" in my Pickup macros so I can tell when they execute. I can confirm that it's not making it to the Pickup macro, but the Debug log shows an M568 P0 A2 which tells me it's executing the tpre0.g file. I cannot say as of yet if it's making it to tpost0.g. Will test further when I get a moment.

dc42

@Gamefanatic3D from your M122 report I can see that the job is held up waiting for the tpre file to complete, but it hasn't completed because there are three G10 P0 commands pending. These appear to be coming from your tconfig0.g file.

I don't yet know what is causing these commands to remain pending; however if you insert a M400 or G4 P0 command before the M98 P"/sys/tconfig0.g" line at the end of tpre0.g, that may work around the problem.

Gamefanatic3D

@dc42 Wow, thank you for the second pair of eyes on this. Yes, it was stuck in the tconfig0.g. Putting the M400 just before the M98 command to load my configs is working!

dc42

@Gamefanatic3D thanks, I'm glad that workaround worked for you. I've created https://github.com/Duet3D/RepRapFirmware/issues/998.

dc42 created this issue in Duet3D/RepRapFirmware

closed Queued commands not executed causing tool change to hang #998

Gamefanatic3D

@dc42 Let me know if you need anything from me. I realize I am probably apart a small few experiencing this issue.

T3P3Tony

@Gamefanatic3D please will you test with 3.6beta3:
https://forum.duet3d.com/topic/37289/software-version-3-6-0-beta-3-now-available

Gamefanatic3D

@T3P3Tony I'm in the process of converting that machine from a Duet2 Wifi to Duet3. I'm conceding I've gone beyond the limits of a Duet2 and looking to move to 5 tools, plus filament box sensors and possibly even filament changers. I'll test once I get it FoC, but I can't guarantee when that will be with my hectic schedule.