Movement after M112 E-Stop!

[hpiz]

So idk if this is expected behavior, but if it is, I'm gonna have to sell my 6XD and get a Mach3 at this point...

I'm trying to make a ridiculously dangerous CNC machine safe, this is not specific to CNC use cases however.

My problem is that M112 does not actually stop my machine, sure it halts for a second, and then it seems to continue reading the buffer for more moves, somehow reactivates the alarmed driver and continues moving for a second and then finally stops.

This is absolutely terrible for a dual drive axis like my CNC gantry. I already broke my prototype 3d printed brackets because of this, and once those are milled from aluminum, it will cost hundreds of dollars if this happens because those aren't what will break when it's all metal. Admittedly, I have only been focusing on my dual drive axis and have not tested my other single drive axes, but this is bad enough to not proceed any further.

As mentioned, I have alarm inputs all set up and a dual drive Y axis gantry, although the behavior is identical regardless of how M112 is triggered...

This is my driver-error.g which does execute when exected...

M291 R"Driver Error!" P{"Driver " ^ param.B ^ "." ^ param.D ^ " threw an error."} T0; 
M112

I even tried the following with no change in behavior...

M291 R"Driver Error!" P{"Driver " ^ param.B ^ "." ^ param.D ^ " threw an error."} T0; 
while {iterations<101}
   M112
   G4 50

and the following link is a video I took while grabbing one of the Y axis drive belts to over-torque the servo motor to cause an alarm while running a macro movement loop...

https://drive.google.com/file/d/1e4E6Nwl_8v_Z-Q1TkidRHHuBT-Pesv_H/view?usp=sharing

I love Duet for 3d printers, but I really am doubting the safety of using a Duet for this application, should I just buy a proper CNC controller?

Note: I will have a manual e-stop that will cut power, but some stops need to happen automatically.

[hpiz]

@hpiz Oh and I'm running the latest 3.4.6 firmware

[deckingman]

@hpiz That's unusual behaviour, not normally seen. As it's a 6XD, I'm just wondering if the external drivers are "caching" the movement queue. I.e, perhaps the board stops current moves and empties the movement queue but can't do anything about the queued moves that have already been sent to the external drivers. Dunno - just a guess. No doubt someone more knowledgeable that I will jump in.

Edit. IMO, you can't beat a "proper" emergency stop that cuts power.

[timschneider]

@hpiz
Can you try to remove the M291, as this causes RRF to read the file line by line. Maybe the movement queue is stopped for the driver-error event (first stop in your video), queues the Message (movement after first stop, as there are other queues with movement commands), executes M112 (second and final stop).

So that your driver error only contains 1 line, with only the M112 in the first and only line of that file.

@hpiz said in Movement after M112 E-Stop!:

while running a macro movement loop...

another try would be, to do the movement loop inside a actual job-file, e.g. some sort of .gcode

[hpiz]

@deckingman That's more or less what I'm thinking is going on. It would appear the duet driver chips run independently from the MCU but share a memory buffer, the drivers "reserve" a couple commands at a time, and when M112 comes the MCU can only remove "unreserved" buffer memory, leaving one move in the buffer, as it does actually stop the currently running move.

[hpiz]

@timschneider I will try that, but it's effectively already ruled out, the same behavior happens regardless of how I call M112 (DWC EStop button on web, wired estop button set up as estop trigger, or driver error inputs), even with no M291 being called.

[hpiz]

@timschneider Also, this is an ATC router, I will be utilizing macros heavily, if saying "don't use macros" is the answer, the duet is not up to this job.

[hpiz]

To help explain why this is so important, my Y axis gantry will weigh 55kgs attached with 8mm x 35mm bolts with half inch aluminum brackets, if one side stops while doing literally anything, and the other side keeps moving, terrible terrible things will happen (and that's exactly how my plastic prototype parts got torn apart when the gantry was still attached and I tested a wired estop button while in a movement loop). In no circumstance can the two Y axis drives be out of skew more than an inch or so.

It also can't just move both sides after calling for a stop, it doesn't maintain position on the "move after M112" and can go, both outside axis limits, or at best, drive the spindle into unmilled stock with the spindle stopped.

[timschneider]

@hpiz said in Movement after M112 E-Stop!:

@timschneider Also, this is an ATC router, I will be utilizing macros heavily, if saying "don't use macros" is the answer, the duet is not up to this job.

I think the answer is not, don't use macro. The answer is more or less, if you want duet to stop, just use one G-Code M112 in a trigger or event file and that M112 must be placed at the beginning of the file on the first line without any comment in front of it, otherwise the command my be delayed or the behaviour may change.

btw. my setup is quite similiar to yours. The weight of the machine is about 300kg and the gantry is about 50-60kg, with everthing attached. I use an additional safety relais from omron for the spindle and the duet e-stop.

[timschneider]

@hpiz
Do you use M669 S and T ?

Snnn Segments per second (RRF 3.3 and later)
Tnnn Minimum segment length (mm) (RRF 3.3 and later)
In RRF 3, segmentation is not used unless the S and/or T parameter is given. Segmenting moves is useful when faster pause response is wanted.

I use S2 T1.

[hpiz]

@timschneider Thank you for helping me out with this, your machine is on another level, superb job on it.

So I tried removing everything but M112 from driver-error.g

Same behavior.

I didn't even have an M669 in my config, and after adding it with M669 K0 S2 T1 it stopped properly. I will have to test a lot more times, but it's got me feeling a lot better. Does setting this segmentation cause slow arcs or jittery movement while cutting?

Idk if the behavior I was having should happen without setting S and T for M669, I mean it was doing it even with the DWC web estop "button"

An example video before adding M669 K0 S2 T1...
https://drive.google.com/file/d/1eZz5nEXnwz80k7NNszTq5wNqKv8INda2/view?usp=drive_link

Regardless, I see you using a duet on that amazingly awesome machine and I know I should be able to do the same with my humble router.

I really hope I can tune the torque limits on my servo motors to not falsely trigger but also not be able to tear itself apart, but I think the mechanical advantage may be too high to control that finely.

[hpiz]

@timschneider So it's better, but still has moments when it fails, had the same behavior 2 out of 6 tests. Unfortunately still not good enough...

Video of one of the two failures to stop with M669 K0 S2 T1...
https://drive.google.com/file/d/1f8hx3Qj4nXPbtJ93OoA9ezyFlq1tjeU4/view?usp=sharing

[sinned6915]

@hpiz i am literally struggling with today too....

I think that Duet for a router or CNC is fine. the problem is we need to config EVERYTHING.

i was thinking that my estop button will cut motor power only, not controller power.

problem is, i think that the system will still try to move the motors with the 5V controller input power.

[hpiz]

@sinned6915 I'm not really worried about my manual estop button, that will cut all power via relays, this is for situations like a power supply failure, over-torque, or lost steps, things I can't detect with my 5 human senses, let alone react fast enough to.

And I keep coming back to, there is no config for M112, it should just stop everything.

[timschneider]

@hpiz
Maybe it is some kind of a bug.

can you enable event logging? Post your config.g, your movement macro and describe your setup in more detail. Do you use the Enable Pin on your servo controllers?

M929 P"0:/sys/eventlog.txt" S3

and reproduce the error.

If your movement macro looks something like

while true
  G1 X10 F20000
  G1 X600 F20000

change it to:

while true
  G1 X10 F20000
  G1 X600 F20000
  G4 P500

as the first example will cause the duet to have 100% system load and the sending of the M112 may fail or is delayed.

Maybe @dc42 can help us with that, as I don't understand the first stop and the second and final stop. What is causing the first stop? as it is not the M112.

EDIT: I thought somehow that he is in SBC mode.

[gloomyandy]

Are you running standalone or with an SBC? As above more details of your configuration will probably help folks investigate further.

[hpiz]

@timschneider and @gloomyandy Sounds like I've got some homework today, yes, I will gather all information, logs, and run the suggested tests and modifications, until then I can certainly fill in what the setup is...

• Running Duet 6XD with no SBC

• Utilizing enable signal on all drives, currently active disabled/low (plan to reverse in future for safety)

• Utilizing error signal on all drives, currently active enabled/high (again plan to reverse for safety, especially when 3.5.1 comes out with polarity configuration per drive for error signals)

@timschneider said in Movement after M112 E-Stop!:

Maybe @dc42 can help us with that, as I don't understand the first stop and the second and final stop. What is causing the first stop? as it is not the M112.

My understanding is that M112 is executed on the first stop, the DWC even immediately shows the Estop message and darkens the webpage (message can almost be seen in https://drive.google.com/file/d/1f8hx3Qj4nXPbtJ93OoA9ezyFlq1tjeU4/view, but you can see the webpage darken on first stop)

@timschneider said in Movement after M112 E-Stop!:

Maybe you hit a case where the setup is not getting any low latency.

The first stop is always very low latency, mere milliseconds. Unless that stop is somehow unrelated to M112 then it's not taking too long to trigger for my needs.

• Config.g was posted earlier and is still current

• I don't intend to use loops in macros really, this is just the best movement test I could come up, I will try and get an actual job running as well and test

• The movement macro loop used in the above posted videos is

G91
while {iterations<5}
   G1 Y-100 F4000
   G1 Y100 F4000

• A dwell in the loop would not be representative of an actual job (even if it works, it wouldn't prove that a real job will stop) so I think I will try...

G91
G1 Y-100 F4000
G1 Y100 F4000
G1 Y-100 F4000
G1 Y100 F4000
G1 Y-100 F4000
G1 Y100 F4000
G1 Y-100 F4000
G1 Y100 F4000
G1 Y-100 F4000
G1 Y100 F4000

• Macros will be used for automatic tool changing and accessories like mist coolant, touch probes, etc

• I'm just a measly sys admin by day so these embedded systems are somewhat foreign to me in terms of their system architecture and data flow. But I feel like the focus on the driver error signal is not warranted. As I've shown in video, this can happen regardless of the means by which M112 is sent (I just like grabbing the drive because it's my primary objective, as I will not utilize M112 to manually stop the machine, will use button to relays for power cut).

When I have logging and more answers to my "homework questions" I'll be sure to post them.

[hpiz]

@timschneider Here is the latest test run configuration that started moving again after clicking the EStop button in DWC in the following macro...

Y Movement Loop

G91
G1 Y-100 F4000
G1 Y100 F4000

G1 Y-100 F4000
G1 Y100 F4000

G1 Y-100 F4000
G1 Y100 F4000

G1 Y-100 F4000
G1 Y100 F4000


G1 Y-100 F4000
G1 Y100 F4000

with the following config.g

; Configuration file for Duet 3 MB 6XD (firmware version 3.3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.3.16 on Wed Jul 26 2023 17:05:34 GMT-0400 (Eastern Daylight Time)

; General preferences
G90                             ; send absolute coordinates...
M550 P"JP-60120"                ; set printer name
M929 P"0:/sys/eventlog.txt" S3  ; Enable debug logging

;CNC Mode
M950 R0 C"out8+out5" Q20 L24000 ; Create spindle index 0, with PWM pin on heater 3 and 12000 RPM achieved at full PWM
M563 P1 S"Spindle 1" R0         ; Create tool 1 with spindle 0 and call it "Spindle 1"
M453                            ; Select CNC device mode
M669 K0 S2 T1


; CNC Pendant Enable
M575 P1 S1 B57600               ;Enable serial comm on io0 56700 baud rate

; Set up outputs
M950 P0 C"out4" ; Define tool loosen pin as P0 mapped to output 8
M950 P1 C"out3" ; Define chip blast pin as P1 mapped to output 7
M950 P2 C"out7" ; Define mist coolant pin as P2 mapped to output 6
M950 P3 C"out6" ; Define touch blast pin as P3 mapped to output 4

; Set up inputs
M950 J0 C"io1.in" ;Define tool probe depressed over limit from io3 to Pin 0
M950 J1 C"!io8.in.iso" ;Define VFD Fault input from io8 to Pin 1

; Network
M552 P0.0.0.0 S1                ; enable network and acquire dynamic address via DHCP
M586 P0 S1                      ; enable HTTP
M586 P1 S0                      ; disable FTP
M586 P2 S0                      ; disable Telnet

; Drives
M569 P0.4 S0 T3:3:6:0       ; physical drive 0.4 goes backwards
M569 P0.5 S0 T3:3:6:0       ; driver 5 requires an active high enable, 2.5us minimum step pulse, 2.5us minimum step interval, 5us DIR setup time and no hold time
M569 P0.3 S0 T3:3:6:0       ; driver 5 requires an active high enable, 2.5us minimum step pulse, 2.5us minimum step interval, 5us DIR setup time and no hold time
M569 P0.0 S0 R0 T5:5:9:0       ; physical drive 0.0 goes backwards
M569 P0.1 S1                    ; physical drive 0.1 goes forwards
M569 P0.2 S1                    ; physical drive 0.2 goes forwards
M584 X0.3 Y0.4:0.5 Z0.0             ; set drive mapping
M92 X160.00 Y320.00 Z640.00     ; set steps per mm
M566 X1800.00 Y180.00 Z120.00     ; set maximum instantaneous speed changes (mm/min)
M203 X15000.00 Y12000.00 Z5000.00 I5  ; set maximum speeds (mm/min)
M201 X400.00 Y200.00 Z200.00        ; set accelerations (mm/s^2)
M564 H0                         ; Allow stepper movement before homing
; Axis Limits
M208 X0 Y0 Z0 S1                ; set axis minima
M208 X600 Y1200 Z170 S0         ; set axis maxima

; Endstops
;M574 X1 S1 P"io0.in"            ; configure switch-type (e.g. microswitch) endstop for low end on X via pin io0.in
M574 Y2 S1 P"!io2.in+!io4.in"   ; configure switch-type (e.g. microswitch) endstop active low for high end on Y via pin io5.in and io2.in
M574 Z2 S1 P"!io8.in"            ; configure switch-type (e.g. microswitch) endstop for high end on Z via pin io2.in

; Z-Probe
M558 P5 C"io3.in" H5 F120 T6000 ; set Z probe type to switch and the dive height + speeds

;Triggers
M581 T2 P0 S1 ; Run trigger2.g on pin 0 active->inactive for tool probe overpress
M581 T3 P1 S1 ; Run trigger3.g (EmStop) when pin 9 goes from active to inactive indicating VFD fault


; Heaters
M140 H-1                        ; disable heated bed (overrides default heater mapping)

; Fans

; Tools
M563 P0 D0 F0                   ; define tool 0
G10 P0 X0 Y0 Z0                 ; set tool 0 axis offsets



;Machine Initialization
M18                          ; disable all steppers
M17                          ; enable z stepper so it doesn't fall

; Custom settings are not defined

and the following eventlog.txt was recorded

power up + 00:00:00 [info] Event logging started at level debug
power up + 00:00:00 [info] Running: Duet 3 MB6XD v1.01 or later: 3.4.6 (2023-07-21 14:11:58)
power up + 00:00:00 [debug] HTTP is enabled on port 80
power up + 00:00:00 [debug] FTP is disabled
power up + 00:00:00 [debug] TELNET is disabled
power up + 00:00:00 [warn] Error: Pin 'io8.in' is not free
power up + 00:00:00 [warn] Error: bad drive number
power up + 00:00:00 [warn] Error: in file macro line 74: G10: Invalid tool number
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 MB6XD is up and running.
power up + 00:00:10 [warn] Ethernet running, IP address = 10.0.0.210
power up + 00:00:10 [warn] HTTP client 10.0.0.10 login succeeded
2023-10-29 16:08:29 [warn] Date and time set at power up + 00:00:10
2023-10-29 16:09:21 [warn] HTTP client 10.0.0.214 login succeeded
2023-10-29 16:09:21 [warn] HTTP client 10.0.0.214 login succeeded
2023-10-29 16:09:22 [warn] HTTP client 10.0.0.10 login succeeded
2023-10-29 16:09:23 [warn] HTTP client 10.0.0.221 login succeeded
2023-10-29 16:10:21 [info] Event logging stopped

Other than letting me know I have a total of 3 different computers in my house with DWC up in the browser, and confirming my theory that opto-isolated inputs are shared with direct inputs as I was feeling was the case, but is not documented clearly (note: Z axis endstop is not even wired yet), it doesn't reveal much.

I didn't record this one, because the behavior is identical to the ones I've already posted. This is feeling more and more like a very low level firmware bug.

I was out all day at a machine shop trying to mill some parts needed for my machine (replacing the plastic carriage brackets for gantry), so I didn't and probably won't have time today to test a proper G code job, but I suspect much the same, and also doesn't make it happening in a macro ok.

[timschneider]

thank you for the details.

@hpiz said in Movement after M112 E-Stop!:

M564 H0 ; Allow stepper movement before homing

can you try to remove this from your config and instead set a homing for Y manual via M92 Y0 and try again?

And do you use some kind of input filter on your servo drives? or are they using some sort of input filter by default?

Because the M112 command is doing different things:

• it will disable all drivers (disable the enable pin)
• will erase all movement commands from the output-queue (also from the input queue?)
• stop generating Platform Ticks
• reset gcode

[hpiz]

@timschneider So these are my first experience using servo motors so I'm not really well versed with them, but I believe there is no input filtering currently set on them as per the highlighted setting in the following image link...
https://drive.google.com/file/d/1f_pghiU0pyJKdWfWDeJs9BGFsF7hdGVT/view

Ok, so I've commented the M564 out so current config.g is...

; Configuration file for Duet 3 MB 6XD (firmware version 3.3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.3.16 on Wed Jul 26 2023 17:05:34 GMT-0400 (Eastern Daylight Time)

; General preferences
G90                             ; send absolute coordinates...
M550 P"JP-60120"                ; set printer name
M929 P"0:/sys/eventlog.txt" S3  ; Enable debug logging

;CNC Mode
M950 R0 C"out8+out5" Q20 L24000 ; Create spindle index 0, with PWM pin on heater 3 and 12000 RPM achieved at full PWM
M563 P1 S"Spindle 1" R0         ; Create tool 1 with spindle 0 and call it "Spindle 1"
M453                            ; Select CNC device mode
M669 K0 S2 T1


; CNC Pendant Enable
M575 P1 S1 B57600               ;Enable serial comm on io0 56700 baud rate

; Set up outputs
M950 P0 C"out4" ; Define tool loosen pin as P0 mapped to output 8
M950 P1 C"out3" ; Define chip blast pin as P1 mapped to output 7
M950 P2 C"out7" ; Define mist coolant pin as P2 mapped to output 6
M950 P3 C"out6" ; Define touch blast pin as P3 mapped to output 4

; Set up inputs
M950 J0 C"io1.in" ;Define tool probe depressed over limit from io3 to Pin 0
M950 J1 C"!io8.in.iso" ;Define VFD Fault input from io8 to Pin 1

; Network
M552 P0.0.0.0 S1                ; enable network and acquire dynamic address via DHCP
M586 P0 S1                      ; enable HTTP
M586 P1 S0                      ; disable FTP
M586 P2 S0                      ; disable Telnet

; Drives
M569 P0.4 S0 T3:3:6:0       ; physical drive 0.4 goes backwards
M569 P0.5 S0 T3:3:6:0       ; driver 5 requires an active high enable, 2.5us minimum step pulse, 2.5us minimum step interval, 5us DIR setup time and no hold time
M569 P0.3 S0 T3:3:6:0       ; driver 5 requires an active high enable, 2.5us minimum step pulse, 2.5us minimum step interval, 5us DIR setup time and no hold time
M569 P0.0 S0 R0 T5:5:9:0       ; physical drive 0.0 goes backwards
M569 P0.1 S1                    ; physical drive 0.1 goes forwards
M569 P0.2 S1                    ; physical drive 0.2 goes forwards
M584 X0.3 Y0.4:0.5 Z0.0             ; set drive mapping
M92 X160.00 Y320.00 Z640.00     ; set steps per mm
M566 X1800.00 Y180.00 Z120.00     ; set maximum instantaneous speed changes (mm/min)
M203 X15000.00 Y12000.00 Z5000.00 I5  ; set maximum speeds (mm/min)
M201 X400.00 Y200.00 Z200.00        ; set accelerations (mm/s^2)
;M564 H0                         ; Allow stepper movement before homing
; Axis Limits
M208 X0 Y0 Z0 S1                ; set axis minima
M208 X600 Y1200 Z170 S0         ; set axis maxima

; Endstops
;M574 X1 S1 P"io0.in"            ; configure switch-type (e.g. microswitch) endstop for low end on X via pin io0.in
M574 Y2 S1 P"!io2.in+!io4.in"   ; configure switch-type (e.g. microswitch) endstop active low for high end on Y via pin io5.in and io2.in
M574 Z2 S1 P"!io8.in"            ; configure switch-type (e.g. microswitch) endstop for high end on Z via pin io2.in

; Z-Probe
M558 P5 C"io3.in" H5 F120 T6000 ; set Z probe type to switch and the dive height + speeds

;Triggers
M581 T2 P0 S1 ; Run trigger2.g on pin 0 active->inactive for tool probe overpress
M581 T3 P1 S1 ; Run trigger3.g (EmStop) when pin 9 goes from active to inactive indicating VFD fault


; Heaters
M140 H-1                        ; disable heated bed (overrides default heater mapping)

; Fans

; Tools
M563 P0 D0 F0                   ; define tool 0
G10 P0 X0 Y0 Z0                 ; set tool 0 axis offsets



;Machine Initialization
M18                          ; disable all steppers
M17                          ; enable z stepper so it doesn't fall

; Custom settings are not defined

I also made a test job file...
1001.g

;1001
;T24  D=10 CR=0 - ZMIN=-1 - FLAT END MILL
G90
G21


G1 Y500 F4000
G1 Y600 F4000

G1 Y500 F4000
G1 Y600 F4000

G1 Y500 F4000
G1 Y600 F4000

G1 Y500 F4000
G1 Y600 F4000

G1 Y500 F4000
G1 Y600 F4000

Same behavior...

And here is the debug log

power up + 00:00:00 [info] Event logging started at level debug
power up + 00:00:00 [info] Running: Duet 3 MB6XD v1.01 or later: 3.4.6 (2023-07-21 14:11:58)
power up + 00:00:00 [debug] HTTP is enabled on port 80
power up + 00:00:00 [debug] FTP is disabled
power up + 00:00:00 [debug] TELNET is disabled
power up + 00:00:00 [warn] Error: Pin 'io8.in' is not free
power up + 00:00:00 [warn] Error: bad drive number
power up + 00:00:00 [warn] Error: in file macro line 74: G10: Invalid tool number
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 MB6XD is up and running.
power up + 00:00:10 [warn] Ethernet running, IP address = 10.0.0.210
power up + 00:00:10 [warn] HTTP client 10.0.0.10 login succeeded
2023-10-30 17:34:44 [warn] Date and time set at power up + 00:00:10
2023-10-30 17:34:44 [warn] HTTP client 10.0.0.221 login succeeded
2023-10-30 17:34:44 [warn] HTTP client 10.0.0.221 login succeeded
2023-10-30 17:34:44 [warn] HTTP client 10.0.0.214 login succeeded
2023-10-30 17:34:44 [warn] HTTP client 10.0.0.214 login succeeded
2023-10-30 17:34:44 [warn] HTTP client 10.0.0.10 login succeeded
2023-10-30 17:36:23 [warn] Error: G0/G1: insufficient axes homed
2023-10-30 17:36:34 [warn] Started printing file 0:/gcodes/1001.gcode
2023-10-30 17:36:34 [debug] File 0:/gcodes/1001.gcode selected for printing
2023-10-30 17:36:48 [debug] Done printing file
2023-10-30 17:36:48 [debug] Done printing file
2023-10-30 17:36:48 [warn] Cancelled printing file 0:/gcodes/1001.gcode, print time was 0h 0m
2023-10-30 17:36:48 [info] Event logging stopped

Then, based on your comment about the servo input filtering, I decided to test both my Y axis servo motors, and my Z which is a stepper...

1001.g with Z movement

;1001
;T24  D=10 CR=0 - ZMIN=-1 - FLAT END MILL
G90
G21


G1 Y500 Z100 F4000
G1 Y600 Z0 F4000

G1 Y500 Z100 F4000
G1 Y600 Z0 F4000

G1 Y500 Z100 F4000
G1 Y600 Z0 F4000

G1 Y500 Z100 F4000
G1 Y600 Z0 F4000

G1 Y500 Z100 F4000
G1 Y600 Z0 F4000

And in the video below, the stepper also moves after the first stop (though not as far or as long as the servos, not sure if the amount of movement per axis has any impact), so even with steppers, there would still be enough movement to easily break a tool or shove the tool right through my part or bed. (Note: the Z axis stepper is the thing with the yellow tape flag spinning with the shaft)

https://drive.google.com/file/d/1ffUtCYofIbbLlfk8L9m_zjEHs5LbMnmi/view?usp=sharing

I've also noticed I can get pretty much 100% failure rate by triggering an estop near the end of a move.