Tighter control for waiting for motion commands to complete

poofjunior

Greetings,

I have a setup where a python program controls multiple devices on a liquid/labware handling platform. This setup has multiple tools, controlled with Raspi GPIOs, while a connection to DSF directs motion through a Duet3.

I'm struggling to find a de-facto interface that reliably tells me the state of the machine following a recently issued command such that subsequent tool commands don't execute prematurely.

My original setup looks like this: I have two threads. The main thread connects to DSF in COMMAND mode for issuing GCode. The second thread connects to the machine in SUBSCRIBE mode and is setup to update the local copy of the machine state every 0.4 seconds.

In the original setup, tool commands that follow motion commands are being executed before the motion related commands have been completed. That's the case because it takes up to 0.8 seconds after issuing a motion command for the machine state to register it and change the status from IDLE to BUSY.

Here's some print statements from the console

Moving to: (96.167, 79.750) # GCode has been sent across the socket
start state: idle | 1858.664
end state:   idle | 1858.967
start state: idle | 1859.085
end state:   idle | 1859.367
start state: idle | 1859.473
curr state:  busy | 1859.767 # Status now changes from idle to busy
curr state:  busy | 1860.167
curr state:  busy | 1860.568
curr state:  busy | 1860.967
curr state:  busy | 1861.367
curr state:  busy | 1861.767
end state:   idle | 1862.167

Then I rewrote my setup, adding an INTERCEPT connection to the main thread in EXECUTED mode. When motion commands are issued, this INTERCEPT immediately waits for a response that the command was executed, after which it issues a resolve command. I would think that, at this point, the machine status would now be busy at this point, but that is not the case.

Here's some console output of the result.

sending: {'code': 'G0 Z50.0 F13000', 'channel': 0, 'command': 'SimpleCode'}
(INT) received:
{'channel': 'HTTP',
 'command': 'Code',
 'comment': None,
 'filePosition': None,
 'flags': 6,
 'indent': 0,
 'keyword': 0,
 'keywordArgument': None,
 'length': 16,
 'lineNumber': None,
 'majorNumber': 0,
 'minorNumber': None,
 'parameters': [{'isString': False, 'letter': 'Z', 'value': '50.0'},
                {'isString': False, 'letter': 'F', 'value': '13000'}],
 'result': [{'content': '',
             'time': '2020-08-20T21:34:33.3198983+01:00',
             'type': 0},
            {'content': '\n',
             'time': '2020-08-20T21:34:33.321111+01:00',
             'type': 0}],
 'sourceConnection': 85,
 'type': 'G'}

(COM) received: {"result":"","success":true}
curr state:  idle | 12266.357
curr state:  idle | 12266.666
curr state:  busy | 12267.066
curr state:  busy | 12267.466
curr state:  busy | 12267.866
curr state:  busy | 12268.266
curr state:  busy | 12268.666
curr state:  busy | 12269.066
curr state:  busy | 12269.466
curr state:  busy | 12269.866
curr state:  busy | 12270.266
curr state:  busy | 12270.666

Same problem. There is still a 0.4-0.8 second wait where the machine still returns idle before it switches back to busy to handle the movement command.

Ok, 2 questions now:

am I using Intercept connections in Execute mode wrong? From the docs, I would think that waiting for a resolved command to be executed means that it's now in the queue to be processed. That would also make me think that the machine status would update to BUSY in that time so that the next time I poll it I'd get a busy.
Are there any examples out there of using DSF to issue commands, wait for their completion, and then issue subsequent commands upon completion? If not, I'm happy to work back and forth to make this a definitive example.

The workaround

In my current setup, I'm currently tracking machine position and then blocking further execution until the machine position reflects the desired position. That way I'm sure that subsequent tool commands don't get issued early. But this seems like a hack since I need to bookkeep machine position separately in Python too, and DSF already does a good job of this.

Also, migrating from the github issue.

dc42

What are the "subsequent tool commands" that you are referring to?

Are you aware of the M400 command?

poofjunior

Whoop, the subsequent tool commands are specific to my python application, but they basically look like this:

 self.move_xy_absolute(x,y) # Position over the well at safe z height.
 self.move_xyz_absolute(z=plunge_height) # Plunge the sonicator tip into the media
 self.sonicator.sonicate(seconds)

The first two lines eventually reduce down to a couple of "G0" commands sent over the socket in Command mode, while "sonicate" is built on top of Adafruit's adafruit_mcp4725 library to control DAC via I2C.

    def sonicate(self, exposure_time: float = 1.0, power_percentage: float = 0.4):
        """enable the sonicator at the power level for the exposure time."""
        self.dac.normalized_value = power_percentage
        self.sonicator_enable.value = True
        time.sleep(exposure_time)
        self.sonicator_enable.value = False
        self.dac.normalized_value = 0

What I'm looking for is a definitive way of knowing that the Duet is done executing the final G0 such that I don't invoke the sonicator before the Duet is done moving. The current issue I'm facing is that there's a non-negligible delay between sending a gcode command and getting an updated machine model packet that represents that the command is being processed. Here I'm waiting for the machine model to change from "idle" to "busy" in the machine "status" field. But I'm observing that it takes up to a second for the machine model to actually change its status. The result is that polling the machine status for "idle" may return an "idle" before the machine has even started moving. Adding an M400 doesn't actually fix this initial delay.

Because two distinct operations in the world are happening on separate interfaces controlled in python, I need a way to guarantee that the machine has processed and completed a particular GCode from the DSF API before having my python application invoke the next command.

Let me know if this still doesn't make sense, and I'd be happy to add more detail! Thanks for taking a look!

If you're curious, all of this code is in a public repo here, but it could use some light polishing.

chrishamm

So just to get this right: You want to move to a certain position in a macro file, call some arbitrary code to switch something on/off and then resume movement? If yes, choose an intercepting connection, choose your own G/M-code (e.g. M1000) and do the following

in your macro:

G0 X<pos> Y<pos> ; move to the position
M400 ; wait for pending moves to finish
M1000 ; do something on the Pi

in your interceptor (Pre or Post mode) in your Python script check if the code is M1000. If it isn't, tell DCS to ignore the code. If it is M1000:

- Flush the corresponding G-code channel and check if it returns true. If it doesn't, tell DCS to cancel the code
- Do whatever you need to do (turn the GPIO on/off)
- Send arbitrary codes using the same intercepting connection if you need to (yes, that's possible)
- Resolve the code being intercepted with a result of your choice (an empty one works too)

RRF and DSF buffer codes internally to achieve acceptable throughput when lots of small moves are processed so the flush command is mandatory. If you do not want to use M400, you may lock/unlock the movement system in RRF too, but I suppose M400 is sufficient for your use-case.

Also, as you noticed, the object model synchronization is NOT real-time as it is polled in configurable intervals and it should not be used to determine the position if you're waiting for a certain position to be reached.

poofjunior

So just to get this right: You want to move to a certain position in a macro file, call some arbitrary code to switch something on/off and then resume movement?

That's mostly right! I wasn't planning on doing this from a macro, just by simply sending a stream of GCodes on the socket connection.

Huh, this is a clever solution, and I might play around with it. But, now that you mention that the object model synchronization isn't real time, it may be that I'm simply building this application on top of the wrong interface. Something as simple as getting an "OK" from an M400 from an interface that invokes the motion planner would be totally sufficient.

Thanks for pointing me in the right direction!