Movement after M112 E-Stop!

hpiz

@timschneider When I get home from work today I'll grab that video evidence, but it does the same thing, albeit sometimes heavily delayed (~3-4 seconds), due to my WiFi I think, I've been having some intermittent disconnect issues with it lately.

@dc42 would know best I think, but is it possible there is a need to either reflash my firmware, or possibly an issue with the board (I find the latter hard to believe).

timschneider

@hpiz maybe a dumb question, but do you have a while loop in your daemon.g without a G4?

I try to understand what is causing the delay, as dc42 said, if the M112 is processed, there should be no delay. So something is maybe delaying the processing of the M112, but again that is not explaining the first and the second stop. That was the reason why I asked for the power supply (short power down/undervoltage can cause many different sotware problems), but I think this is sufficient. The reason why I keep asking about the while loops without a "break" is, that the control of the gcode processing is not handed back to process other codes if it is running in a tight loop. Or if for what ever reason there is a blocking(queueable) G-Code infront of the M112, the processing of the M112 will be delayed e.g. turn a relay off with M42 (that was the reason why I asked you to put the M112 at the first and only position of the driver-error.g).

On last idea is, to wire an e-stop and configure it as a estop in duet - just to rule the gcode processing out. if this also failes, it is something in the movement system or the drivers itself, but I don't think it is related to the drivers cause you are using two different kinds of drivers (servos and steppers)

e.g. recycle your tool probe depressed over limit and uncomment T2 under triggers

M581 T0 P0 S1 R0

hpiz

@timschneider Here's a test without the additional modifications you suggested using the EStop button in DWC...

https://drive.google.com/file/d/1hTqF0GCi4dEQjOZyk1uR_Ctzywv-66Zg/view?usp=drive_link

I really think the delay I mentioned is just a network communication delay and not a delay within the Duet itself (it's not delayed every time), but I could be wrong. I would rate this test as having zero delay, as soon as I hit the button the first (very short) stop happened "instantly".

@timschneider said in Movement after M112 E-Stop!:

maybe a dumb question, but do you have a while loop in your daemon.g without a G4?

I don't think any question about this behavior could be considered dumb, but no, I haven't even made a daemon.g, the only configuration I've done so far is the config.g, a few custom M command system macros, and the driver-error system macro everything else is stock.

Screenshot of Duet System Files
^Any file with a modify date of 7/26/2023 is unmodified. Everything other than config.g is very very simple, but let me know if I need to share the contents of any of them.

@timschneider said in Movement after M112 E-Stop!:

I keep asking about the while loops without a "break" is, that the control of the gcode processing is not handed back to process other codes if it is running in a tight loop.

I agree completely, it's what I would assume first as well, that would presumably have the same effect I'm having, but I only ever had the one loop in my movement test macro, no other loops were defined by me anywhere else.

And finally, I changed the tool probe overpress trigger to T0 and got the same behavior...

https://drive.google.com/file/d/1hVCYuPwHx6Yzlz80FHZKv9pIseA7QbPQ/view?usp=sharing

timschneider

@hpiz thank you for all these test and based on these tests I think, we don't need to focus on the driver-error or external driver configuration. It appears to me, that there is a problem with the following configuration

Duet 3 Main Board 6XD standalone mode
Emergency Stop (DWC and external trigger confirmed)
Active disabled/low external drivers

So may I ask you to do two more tests?

check if the stepper motors are active or disabled after the final stop, are you able to move them freely or not? If they are blocked/active that could describe the first and second stop.
put a M112 in the job file, somewhere in the middle. If the Y-axis is moving after the M112 in the job file, than there is a bug in the software, as the M112 is in the same input channel as the movement commands and there is no "normal" way, that movement will appear after the m112.

hpiz

@timschneider So I'm starting to lose my mind, and have begun talking to myself on video (lol, really so I have to type less, but I am starting to get quite lost)

Disclaimer: While my Z stepper can and has moved after a first stop of M112, it is far far less consistent, more than 80% of the time it stops only once as it should (compared to like 2% for the servos). But while the hardware differences clearly are making a behavioral difference, I believe the underlying cause (that I am currently convinced of) is still a problem. I'll get there...

@timschneider said in Movement after M112 E-Stop!:

Active disabled/low external drivers

I'm starting to narrow onto this or something about the enable signal (although the pulse/step signal is still misbehaving imo), and it could be as simple a fix as a wiring change.... maybe, I'm still very confused though. And my gut tells me no wiring change can fix this.

I'm getting confused about the effect of the Enable jumper, I can have it on either position and the drivers are active with no other changes to Duet config or driver parameters.

Also within config.g what does enable high (R1) on a driver mean if the enable pin is actually a ground sink, does that mean enable high (R1) means the transistor is on for the Enable(-) pin meaning it's actually pulled to ground, because that appears to be what I'm seeing. Enable low (R0) keeps my enable wire high since I have it fed by 5V through the optoisolater at the servo. And when I switch to enable high (R1) the enable wire is low, indicating current is passing through the opto on the driver.

@timschneider said in Movement after M112 E-Stop!:

check if the stepper motors are active or disabled after the final stop, are you able to move them freely or not? If they are blocked/active that could describe the first and second stop.

M18 test + EStop drive enable check - Conclusion: Stepper and servo motors are disabled after final stop

@timschneider said in Movement after M112 E-Stop!:

put a M112 in the job file, somewhere in the middle.

M112 mid job - Conclusion: Same effect

Oh, I didn't stop there (warning: the oscilloscope will make an appearance)

M112 in job with M291 right after - Conclusion: M291 is not executed but G movements are

M112 at end of job expecting no strange movement - Conclusion: Still moved after a stop...

Investigating that last test a little deeper - Conclusion: Yup, it did still move, and past where it should have. Ok, obviously the buffering of move commands is superseded by the M112 which executes as soon as it's read.

What about the Z stepper, it does behave somewhat better - Conclusion: I'd say you can see a few steps after the first pause, but much better than my other drives.

Ok let's test the x axis - Conclusion: You can see a clear stop and restart, then final stop just like the Y. While I say in-vid it's about the servo's, I'm not sure they are to blame, there's more.

I kept going off video to try reversing the enable polarity in both the servo params and the duet, I found some quirks I mention in the next vid but it's really not in my focus right now. Anyways, none of that changed anything about the behavior.

At this point I felt I had no option but to pull out the scope. I'm analyzing the following video with the following assumptions...

There is no signal being generated by the driver, the signals seen on scope are entirely from the Duet, I feel that's a pretty safe bet.
The difference in behavior between my Servo's and my Z stepper is down to some kind of buffer/safe time between disabling the drive and re-enabling. I would have to either also assume most drivers have said wait time, or my board is acting differently.

Watching the control signals on scope in realtime

After this test, I really can't blame the servo's at all. They're doing exactly what they are being told to do by the Duet, think it might be time give up. Why would there be steps still coming out of the duet after a stop? Why would the enable signal go to disable->enable->disable? I really am having a hard time trying to think of any configuration or parameter I can change to just have the Duet not send steps after an M112 let alone deactivate, then reactivate a driver.

hpiz

@hpiz I don't have the energy today, but I plan to show the same signals going to my z driver, but being ignored by the driver, which I assume will be the case. I mean why would one drive output behave differently from another?

timschneider

@hpiz Thanks again for all these tests and measurements that go far beyond what I would have asked for.

I guess I can't help you any further and @dc42 needs to take over, as it appears to me, that there is a problem with the following configuration:

Duet 3 Main Board 6XD standalone mode
CNC Mode
Firmware 3.4.6
Emergency Stop (DWC, gcode (M112 even within the jobfile itself) and external trigger (trigger0) confirmed)
Maybe: two motors on one axis

Observed behaviour:

movement commands are queued and the Movement System starts to move the drives
M112 is processed
all drives are disabled
at least one movemend command is still being processed in the queue and this will reactivate the drives again
all drives are disabled again (is it done by the M112 or maybe a stop.g/cancle.g?)

maybe I can reproduce it with a simple bench setup.

timschneider

@dc42 I was able to reproduce this behaviour with an brand new of the shelf 6XD WD32096 v1.01 firmware 3.4.6 reflashed to be sure. In standalone mode.

Enable Jumper in active Enable.

Following config.g

; Enable network
M552 P0.0.0.0 S1
    
;CNC Mode
M453                            ; Select CNC device mode
M669 K0 S2 T1

M569 R0 P0.4 S0 T3:3:6:0       ; physical drive 0.4 goes backwards
M569 R0 P0.5 S0 T3:3:6:0       ; driver 5 requires an active high enable, 2.5us minimum step pulse, 2.5us minimum step interval, 5us DIR setup time and no hold time
M569 R0 P0.3 S0 T3:3:6:0       ; driver 5 requires an active high enable, 2.5us minimum step pulse, 2.5us minimum step interval, 5us DIR setup time and no hold time

M569 P0.0 S0 R0 T5:5:9:0       ; physical drive 0.0 goes backwards
M569 P0.1 S1                    ; physical drive 0.1 goes forwards
M569 P0.2 S1                    ; physical drive 0.2 goes forwards

M584 X0.3 Y0.4:0.5 Z0.0             ; set drive mapping

M92 X160.00 Y320.00 Z640.00     ; set steps per mm
M566 X1800.00 Y180.00 Z120.00     ; set maximum instantaneous speed changes (mm/min)
M203 X15000.00 Y12000.00 Z5000.00 I5  ; set maximum speeds (mm/min)
M201 X400.00 Y200.00 Z200.00        ; set accelerations (mm/s^2)

;M564 H0                         ; Allow stepper movement before homing
; Axis Limits
M208 X0 Y0 Z0 S1                ; set axis minima
M208 X600 Y1200 Z170 S0         ; set axis maxima

;Machine Initialization
M18                          ; disable all steppers
M17                          ; enable z stepper so it doesn't fall

test_movement.gcode

G90
G21
 
 
G1 Y500 F4000
G1 Y600 F4000
 
G1 Y500 F4000
G1 Y600 F4000
 
G1 Y500 F4000
G1 Y600 F4000
 
G1 Y500 F4000
G1 Y600 F4000
 
G1 Y500 F4000
G1 Y600 F4000

M122
Screenshot 2023-11-03 at 12-03-32 My Duet.png

EmergencyStop pressed mid job via DWC

Channel 1 (yellow) is STEP-
Channel 2 (pink) is ENABLE-

there is a ENABLE low, when the Emergency Stop is pressed, then high again, then low, high, low again (low is off in this setup). I've tried it also the other was around with R1 - it just inverts the curve.

The Setup:

gloomyandy

@timschneider If you still have the test system available, is there any chance you could run the test again but use M112 entered in the console rather than using the emergency stop button. The reason I ask is because the emergency stop button in DWC runs M112 followed by M999 so some of what you are seeing may be down to the software reset (M999) operation.

timschneider

@gloomyandy sure. I did even more tests.
run M112 from console will change the behaviour a bit, the last disable / enable toggle will disappear. It appears to me, that a reset will also toggle the driver enable once.

I've changed the test setup to be able to test the sbc setup.
Unfortunately, it will also fail, so there is also movement after the M112.

the following screens are all taken with SBC setup 3.4.6

Screenshot 2023-11-03 at 13-43-18 rockpi-4cplus.png
and the following config.g

M669 K0 X1:0:0 Y0:1:0 Z0:0:1 S1 T1                      ; Cartesian kinematics

; Drives
M569 P0 R0 T1:1:2:0
M569 P1 R0 T1:1:2:0
M569 P2 R0 T1:1:2:0
M569 P3 R0 T1:1:2:0
M569 P4 R0 T1:1:2:0
M569 P5 R0 T1:1:2:0

M569 P0 S1
M569 P1 S1
M569 P2 S1
M569 P3 S1
M569 P4 S1
M569 P5 S1

G90                                                     ; send absolute coordinates...
M584 X0 Y4:5 Z3                                         ; set drive mapping

M92 X400 Y400 Z800                                      ; set steps per mm 4000 steps/rev / 10 mm pitch - 5mm on z

M566 X120.0 Y120.0 Z120.00 P1                           ; set maximum instantaneous speed changes (mm/min) and apply jerk on every move
M203 X10000.00 Y10000.00 Z7000.00                       ; set maximum speeds (mm/min)
M201 X400.00 Y500.00 Z1000.00                           ; set accelerations (mm/s^2)
M204 P250 T1250                                         ; Set printing and travel accelerations
M84 S30                                                 ; Set idle timeout

; Axis Limits
M208 X0 Y0 Z0 S1                                        ; set axis minima
M208 X1330 Y1036 Z278 S0                               ; set axis maxima

; Mode Selector
M453                                                    ; Select CNC Mode

without M669 S1 T1

with M669 S1 T1

with M112 in console

with Emergency Stop in DWC

hpiz

@timschneider What a relief to hear this is reproducible, I was starting to get really depressed. And you beat me to yanking my 6xd out of the cabinet and wiring resistors on an unconnected drive output on the 6xd. That was literally gonna be my next course of troubleshooting.

I took a vacation day today, so I'm about to test the Z (although your test all but confirm this is board/firmware behavior). Will post vid shortly.

timschneider

I did some more test, as I was not able to see this behaviour in my last cnc build. So I was recording the STEP and Enable Signal for the other driver 0,1,2,3 and 5 as well, all with M112 mapped to the Y-axis.

M584 X4 Y0:1:3:5 Z2
and
M584 X4 Y0:1:2:5 Z3

btw. the resistors are all 15k

driver0

driver1

driver2

driver3

driver5

timschneider

it looks like the motion system takes some time to shut down - so is it possible that the following will reactivate the drive again after an EmergencyStop?

In the function void DDA::Prepare(SimulationMode simMode) noexcept

hpiz

@timschneider Obviously my scope is way worse than Tim's so I would reference Tim's scope grabs over my shoddy videos, but I'm trying to do what I can...

Z stepper test with M112 in job

Z stepper test with no M112 in job, using DWC EStop

And lastly...
Z stepper test using external estop trigger with T0

That last one, in my mind, all but confirms there is a wait delay in that driver after enable is reactivated, and depending on how long the Duet keeps it enabled, eventually it too does move, but reacts to disabling immediately.

timschneider

@hpiz I was using the 6XD quite a while without noticeing, so I think it is a good teamwork!

and I guess I maybe found the place where the delay kicks in

when the seperate move thread was added, the active switch fall overboard and it appears to me, that TerminateAndUnlink needs some time to terminate and unlink, and in the meantime, new moves are prepared, and the prepare move is enableing the drive again.

droftarts

@timschneider @hpiz Thanks for all your investigative work, I've flagged this to @dc42.

Ian

dc42

Hi all, so does this issue occur only when using external drivers, for example with a 6XD? ~~Does it occur only in SBC mode?~~

@timschneider which source code branch are you looking at? The current 3.5-dev and 3.4-dev source code has this for Move::Exit:

void Move::Exit() noexcept
{
	StepTimer::DisableTimerInterrupt();
	rings[0].Exit();
#if SUPPORT_ASYNC_MOVES
	rings[1].Exit();
#endif
#if SUPPORT_LASER || SUPPORT_IOBITS
	delete laserTask;
	laserTask = nullptr;
#endif
	moveTask.TerminateAndUnlink();
}

That should be sufficient to stop all movement.

@hpiz which firmware version are you using?

PS - when Emergency Stop is pressed in DWC, it sends M112 and then after a little while it sends M999 to reset the Duet. At the reset the driver Enable signal may go to Enabled again, depending on its polarity. But by then the movement queue is empty because RRF has re-initialised.

timschneider

@dc42
sorry I was referring to the commit where the Move Task was added.
But you give the right hint, I was looking in the wrong direction, as it can bee seen from the oscilloscope that the stop of the step generation is delayed. So I guess, another code before Move::Exit is delayed or the very first line StepTimer::DisableTimerInterrupt is slow.

I'll be back in the office on monday, then I can do the above tests with Duet 3 6HC.

hpiz is using 3.4.6 - from the debug log

power up + 00:00:00 [info] Running: Duet 3 MB6XD v1.01 or later: 3.4.6 (2023-07-21 14:11:58)

hpiz

@dc42 Hi, I have a Duet 2 I could test 3.4.6 with to see steps on my analog scope. Not sure if that helps...

dc42

All, when you see this behaviour is it just the remainder of the current move that continues to execute after a pause? The Move::Exit code empties the movement queues but I'm wondering whether the step interrupt might be getting re-enabled and perhaps that might cause the current move to complete.