Intercepting Messages from DSF/DCS with dsf-python

davidjryan

Probably one for @Falcounet or @chrishamm ?

How would I go about intercepting messages generated by DSF/DCS with the dsf-python bindings? I have looked at the DWC source on GitHub to try and figure out how it was doing it, but a lot of the coding is beyond me (hence why I am using python!) I just need a slight nudge in the right direction.
https://github.com/Duet3D/DuetWebControl

6HC
Pi5 in SBC Mode
CNC Mode
Custom python app controlling the system
DWC running to provide status and monitoring but system will run if DWC is stopped (if there is contention between my app and DWC for intercepting the messages)

I am trying to capture the messages from DSF/DCS like DWC's Console:

And things like syntax errors, echo strings from meta code, etc. Basically, I want to create my own "Console" within my app so I don't have to use DWC concurrently with my app.

I assume I would create a signal with a dsf-python function that gets raised when a message is sent from DSF/DCS, then have a function to parse the message when triggered?
Or connect via HTTP with:
GET /rr_connect
and the messages will stream through that as they occur until I:
GET /rr_disconnect ?

If able to provide one, a simple code snippet will suffice to get my marbles rolling.

rero

@davidjryan there is a perfect example in the dsf python-bindings:
examples -> custom_m_codes.py

(Sry, don't have the reputation (!) to post links in this forum)

droftarts

@rero You should have enough reputation to post links now. It's an anti-spam measure. I think the link you mean is https://github.com/Duet3D/dsf-python/blob/v3.6-dev/examples/custom_m_codes.py

Ian

davidjryan

@droftarts @rero

Son of a #$@#!

Thanks gents! I'll start playing around with that example.

davidjryan

@droftarts

I have the example running and I'm watching the stream. I have set it to intercept "everything" (removed filters from the InterceptConnection call).

With the example program NOT running, if I execute an G90 G1 X-10 F6000 while X is at 0mm from the DWC Dashboard tab, I get "Error G1: target position outside machine limits" banner message and it goes to the Console. All correct as my X limit is -1mm.

With the example program running, if I execute an G90 G1 X-10 F6000 while X is at 0mm from the DWC Dashboard tab, The "Error G1: target position outside machine limits" banner message does NOT display in DWC nor in the Console, and the example app receives the following:

pi@A1000-0:~/A1000 $ python intercept.py
send: {"mode":"Intercept","version":12,"InterceptionMode":"Pre","Channels":["HTTP","Telnet","File","USB","Aux","Trigger","Queue","LCD","SBC","Daemon","Aux2","Autopause","File2","Queue2","Unknown"],"AutoFlush":true,"AutoEvaluateExpressions":true,"Filters":null,"PriorityCodes":false}
recv: {"success":true}
recv: {"sourceConnection":0,"result":null,"type":"M","channel":"Trigger","lineNumber":null,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":409,"minorNumber":null,"flags":4097,"comment":null,"filePosition":null,"length":null,"parameters":[{"letter":"K","value":"network","isString":true},{"letter":"I","value":"1","isString":false}],"command":"Code"}
1 CDE:
M409 K"network" I1
1 CDE Type:
M
send: {"command":"Flush","Channel":"Trigger","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":43,"result":null,"type":"M","channel":"HTTP","lineNumber":1,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":120,"minorNumber":null,"flags":2049,"comment":null,"filePosition":0,"length":5,"parameters":[],"command":"Code"}
2 CDE:
M120
2 CDE Type:
M
send: {"command":"Flush","Channel":"HTTP","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":43,"result":null,"type":"G","channel":"HTTP","lineNumber":2,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":91,"minorNumber":null,"flags":2049,"comment":null,"filePosition":5,"length":4,"parameters":[],"command":"Code"}
3 CDE:
G91
3 CDE Type:
G
send: {"command":"Flush","Channel":"HTTP","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":43,"result":null,"type":"G","channel":"HTTP","lineNumber":3,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":1,"minorNumber":null,"flags":2049,"comment":null,"filePosition":9,"length":14,"parameters":[{"letter":"X","value":"-10","isString":false},{"letter":"F","value":"6000","isString":false}],"command":"Code"}
4 CDE:
G1 X-10 F6000
4 CDE Type:
G
send: {"command":"Flush","Channel":"HTTP","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":43,"result":null,"type":"M","channel":"HTTP","lineNumber":4,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":121,"minorNumber":null,"flags":2049,"comment":null,"filePosition":23,"length":5,"parameters":[],"command":"Code"}
5 CDE:
M121
5 CDE Type:
M
send: {"command":"Flush","Channel":"HTTP","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":0,"result":null,"type":"M","channel":"Trigger","lineNumber":null,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":409,"minorNumber":null,"flags":4097,"comment":null,"filePosition":null,"length":null,"parameters":[{"letter":"K","value":"network","isString":true},{"letter":"I","value":"1","isString":false}],"command":"Code"}

Switching to InterceptConnection(InterceptionMode.POST gives this:

pi@A1000-0:~/A1000 $ python intercept.py
send: {"mode":"Intercept","version":12,"InterceptionMode":"Post","Channels":["HTTP","Telnet","File","USB","Aux","Trigger","Queue","LCD","SBC","Daemon","Aux2","Autopause","File2","Queue2","Unknown"],"AutoFlush":true,"AutoEvaluateExpressions":true,"Filters":null,"PriorityCodes":false}
recv: {"success":true}
recv: {"sourceConnection":0,"result":null,"type":"M","channel":"Trigger","lineNumber":null,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":409,"minorNumber":null,"flags":4099,"comment":null,"filePosition":null,"length":null,"parameters":[{"letter":"K","value":"network","isString":true},{"letter":"I","value":"1","isString":false}],"command":"Code"}
1 CDE:
M409 K"network" I1
1 CDE Type:
M
send: {"command":"Flush","Channel":"Trigger","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":0,"result":null,"type":"M","channel":"Trigger","lineNumber":null,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":409,"minorNumber":null,"flags":4099,"comment":null,"filePosition":null,"length":null,"parameters":[{"letter":"K","value":"network","isString":true},{"letter":"I","value":"1","isString":false}],"command":"Code"}
2 CDE:
M409 K"network" I1
2 CDE Type:
M
send: {"command":"Flush","Channel":"Trigger","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":0,"result":null,"type":"M","channel":"Trigger","lineNumber":null,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":409,"minorNumber":null,"flags":4099,"comment":null,"filePosition":null,"length":null,"parameters":[{"letter":"K","value":"network","isString":true},{"letter":"I","value":"1","isString":false}],"command":"Code"}
3 CDE:
M409 K"network" I1
3 CDE Type:
M
send: {"command":"Flush","Channel":"Trigger","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":52,"result":null,"type":"M","channel":"HTTP","lineNumber":1,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":120,"minorNumber":null,"flags":2051,"comment":null,"filePosition":0,"length":5,"parameters":[],"command":"Code"}
4 CDE:
M120
4 CDE Type:
M
send: {"command":"Flush","Channel":"HTTP","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":52,"result":null,"type":"G","channel":"HTTP","lineNumber":2,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":91,"minorNumber":null,"flags":2051,"comment":null,"filePosition":5,"length":4,"parameters":[],"command":"Code"}
5 CDE:
G91
5 CDE Type:
G
send: {"command":"Flush","Channel":"HTTP","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":52,"result":null,"type":"G","channel":"HTTP","lineNumber":3,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":1,"minorNumber":null,"flags":2051,"comment":null,"filePosition":9,"length":14,"parameters":[{"letter":"X","value":"-10","isString":false},{"letter":"F","value":"6000","isString":false}],"command":"Code"}
6 CDE:
G1 X-10 F6000
6 CDE Type:
G
send: {"command":"Flush","Channel":"HTTP","SyncFileStreams":false,"IfExecuting":true}
recv: {"result":true,"success":true}
send: {"command":"Resolve","Type":0,"Content":null}
recv: {"sourceConnection":52,"result":null,"type":"M","channel":"HTTP","lineNumber":4,"indent":0,"keyword":0,"keywordArgument":null,"majorNumber":121,"minorNumber":null,"flags":2051,"comment":null,"filePosition":23,"length":5,"parameters":[],"command":"Code"}

So the error response is being redirected away from DWC, but where is it going?

Looking at the responses, I can't decipher if one of those is the error message. They all just look like G and M code intercepts that are confirmation that the DCS received the command. Is there a specific flag or function for errors/faults?

Is the error trapped in the M121 response? Is it the "majorNumber":409 or "flags":4900 keys? If so, is there a lookup table for these values? I didn't see anything on Github.

droftarts

@davidjryan one for @chrishamm , I think!

Ian

rero

@davidjryan sorry I was mislead by your thread-subject and did not carefully read your full post.

I assume you want to check the messages-member of the ObjectModel (see https://github.com/Duet3D/dsf-python/blob/v3.6-dev/src/dsf/object_model/object_model.py and https://github.com/Duet3D/dsf-python/blob/v3.6-dev/src/dsf/object_model/messages/messages.py for details).

I did a quick test with my c#-application and M118- and Plugin-Output arrived successfully ...

Check also https://github.com/Duet3D/dsf-python/blob/v3.6-dev/examples/subscribe_object_model.py - this example shows how to access the ObjectModel.

HTH

PS: Link-reputation achieved!

davidjryan

@rero unfortunately, your assumption is incorrect. I am asking about those messages that are not a part of the object model. I have been capturing the object model for 2 years now with my custom app and I am looking to be able to disable the DWC once I can get the error messages read by my app. I am creating my own "DWC" that is custom to my application (which is not a printer).

The object model messages do not include the "error (RED)" or "warning (YELLOW)" type of messages when there is an overall problem with the system, i.e, any yellow or red message that pops up on the DWC as a banner on the bottom of the DWC webpage or itemized in the Console page/tab as they occur.

From what I can tell, when an error occurs in DSF, the message is "pushed" out there. I need to know where "there" is so I can capture it. I believe the intercept example you and @droftarts sent has me on the right track but I'm still missing a piece of the puzzle.

If I send a non-supported G code from my app, the DWC reports with a yellow warning as an non-supported G command. If I sent a poorly formatted G command, the DWC reports with a red error message with the issue with the command. Neither were sent by DWC yet both were captured as a warning and error by DWC. So something else is at play than just the reading the OM.

Running the intercept example as .PRE or .POST will stop the DWC from reporting those messages. But I can't see within the intercept responses the error message itself unless it is what I mentioned before and there is a lookup table in DWC to textualize the error from the reported code.

rero

@davidjryan if your assumption is correct, two instances/connections/channels of DWC via different browsers should have equal logs.
So if you run a command through the "Send"-function of DWC, the (error-)message will/should be shown everywhere ...

For me that is not the case: every command performed/executed directly reports only to the source-channel which send the command.

For your control-app: what is the answer from the DCS if you perform your test-command?
Like in https://github.com/Duet3D/dsf-python/blob/v3.6-dev/examples/send_simple_code.py ?

If you app is the only controlling-instance, this functionality should be suitable for you ...

chrishamm

@davidjryan That isn't quite right. DWC receives many code replies via the messages array of the object model, at least if they originate from system macros or job files. If you need results of all G-codes (e.g. also from HTTP, USB, or other channels), you should intercept executed G-codes using the IPC API. When you receive them, they should have the result property set which is what you seem to be interested in. Running /opt/dsf/bin/CodeLogger -t executed displays all executed codes, too.

davidjryan

@chrishamm Do you have a code snippet for this? Either C or python?

How is it that DWC gets the messages when it's my app that sends the G or M code that causes the warning or error?

I send some commands via HTTP Request and some commands via Command Request. My app uses PyQt for the GUI and their threading classes so I can run in somewhat of a multitasking environment.

My HTTP request:

class HTTPPostRequestThread(QRunnable):
    """
    Send commands to Duet
    via HTTP request
    """
    def __init__(self, *args):
        super(HTTPPostRequestThread, self).__init__()
        self.url = args[0]
        self.command = args[1]
        self.log = False
        try:
            self.logger = args[2]
            self.log = True
        except:
            pass

    @pyqtSlot()
    def run(self):
        try:
            if self.log:
                self.logger.info(f'Sending {self.command} to DCS via HTTP')        
            result=requests.post(self.url, self.command)
            print(f'HTTP Result = {result}')
        except:
            if self.log:
                self.logger.info(f'Failed to send {self.command} to DCS via HTTP')        
            pass

My CommandConnection request:

class CommandRequestThread(QRunnable):
    """
    Send commands to Duet
    via TCPIP socket
    """
    def __init__(self, *args):
        super(CommandRequestThread, self).__init__()
        self.command = args[0]
        self.log = False
        try:
            self.logger = args[1]
            self.log = True
        except:
            pass
        
    @pyqtSlot()
    def run(self):
        if self.command!='':
            try:
                if self.log:
                    self.logger.info(f'Sending {self.command} to DCS via socket')        
                self.connect = CommandConnection(debug=False)
                self.connect.connect()
                result = self.connect.perform_simple_code(self.command, async_exec=False)
                print(f'Result = {result}')
                self.connect.close()
            except:
                if self.log:
                    self.logger.info(f'Failed to send {self.command} to DCS via socket')        
                pass
        else:
            self.logger.info(f'Command for DCS socket is blank')

Result values:

2025-02-25 08:56:50.964 - INFO - Sending M98 P"configuration/om_faults_reset.g" to DCS via HTTP
HTTP Result = <Response [200]>
2025-02-25 08:56:51.253 - INFO - Tab button (5) Manual pressed
2025-02-25 08:56:52.967 - INFO - Sending set global.bFaultPresent = true to DCS via HTTP
2025-02-25 08:56:52.986 - ERROR - At Least One Axis Is Not Homed
2025-02-25 08:56:52.991 - INFO - Sending set global.bCycleAbort = true to DCS via HTTP
HTTP Result = <Response [200]>
HTTP Result = <Response [200]>
2025-02-25 08:56:56.188 - INFO - okPushButton button pressed
2025-02-25 08:56:58.885 - INFO - Tab button (2) Stock pressed
2025-02-25 08:57:02.301 - INFO - Tab button (7) Recovery pressed
2025-02-25 08:57:06.287 - INFO - pbNegDir button pressed
2025-02-25 08:57:06.299 - INFO - Sending M98 P"functions/jog_axis.g" X"X" Y-1 Z"Neg" to DCS via socket
Result =

Here's the same result but with debug turned on for Command Request:

2025-02-25 09:07:52.514 - INFO - pbNegDir button pressed
2025-02-25 09:07:52.547 - INFO - Sending M98 P"functions/jog_axis.g" X"X" Y-1 Z"Neg" to DCS via socket
send: {"mode":"Command","version":12}
recv: {"success":true}
send: {"command":"SimpleCode","Code":"M98 P\"functions/jog_axis.g\" X\"X\" Y-1 Z\"Neg\"","Channel":"SBC","ExecuteAsynchronously":false}
recv: {"result":"","success":true}
Result = 
2025-02-25 09:07:55.234 - INFO - Sending M98 P"configuration/om_faults_reset.g" to DCS via HTTP
HTTP Result = <Response [200]>

The HTTP request in the above example is just setting globals so the [200] is expected as there are no errors.
The Command request shows no result returned (or success in debug mode). The command for this instance was a G90 G1 X-1 when the axis was not homed, so there was an insufficient axes homed error, which the DWC caught, somehow.

DWC:

chrishamm

@davidjryan I'm sorry but I can't really help with Python scripts. Here the corresponding example: https://github.com/Duet3D/dsf-python/blob/v3.6-dev/examples/custom_m_codes.py Instead of PRE you need to set the interception mode to Executed, then you should be able to inspect the results of executed G/M/T-codes. Of course that plugin needs to run on the SBC itself, you cannot intercept codes remotely without additional infrastructure. Also make sure that your user is part of the dsf group or let it run as root to avoid permission issues while developing.

You can find the source code of the CodeLogger utility here: https://github.com/Duet3D/DuetSoftwareFramework/blob/v3.5-dev/src/CodeLogger/Program.cs See the corresponding wiki -> Third-Party plugins for further details if you want to write a .NET console app.

davidjryan

@chrishamm

I'd like to revisit this question if that's ok. I've been playing around some more and I think I can articulate my issue a little better now. I have downloaded the source for the DWC and DSF. I've skimmed through it but I am certainly not fluent in it.

The messages I am looking to intercept are those that are generated by DSF while it is monitoring the system. These seem to be the "red" banner messages that are displayed at the bottom of the DWC and logged in the DWC console as they occur.

If I execute, via CommandConnection with async_exec = True, from a small app on the Raspberry Pi:
G90 G1 X-10 F1000 (my machine limit for X is 0 and 100)
DWC displays and logs Error: G1: target position outside machine limits
This message is not returned from the CommandConnection call that was used to send the G1 command (obviously since async is set to true). The response is blank which I interpret as the command was validly formatted and accepted by DSF.
If I set async_exec = False, I do get the error message as the result from the CommandConnection call.

So, if async is set to true, DWC gets the error message.
If async is false, my app gets the error message, DWC does not.

So how does the DWC achieve this "capturing" of errors when my app is sending commands in async_exec = true mode? This is what I want to achieve with my larger app; send commands asynchronously and receive error messages as they occur.

Can you point me to the DWC source code where this happens?
I just need a starting point from which I can follow the bread crumbs.

chrishamm

@davidjryan Generic messages come in via the messages[] array of the object model. You need to subscribe to OM changes in order to capture them.

davidjryan

@chrishamm

I've created a function to subscribe to the OM. It sees some of the error messages but not all of them.

From the DWC Dashboard:

When I try to overtravel an axis, the message comes through to my app and the DWC console.
If I try to set a global variable that doesn't exist, the message does not come through to my app but it does show up in the DWC console.
If I try to run a macro that doesn't exist with M98, the message does not come through to my app but it does show up in the DWC console.

The terminal screen on the right shows just the 'message' portion of the OM patch once I receive the full OM on the first call, I didn't want to dump the whole patch on each patch iteration, just when a new message shows up.

The DWC console "sees" everything, my app only saw the target position error.

Does the source of the command call to the DSF matter (my app or DWC) in determining where the message goes?

chrishamm

@davidjryan It really depends on what you actually need to capture. In summary:

G-codes are targeted in SBC mode. If you send a code request, e.g. from DWC, you get back a corresponding reply when the request finishes
This may not apply for codes that are invoked by following macro files. Those messages are output as generic messages (via the OM)
Messages generated from codes that are part of system/trigger macros or print files are sent out as generic messages as well (via the OM)

If you need to capture ALL error messages, you need to intercept ALL codes in "Executed" mode and inspect their result whenever they come in. I suggest you have a look at the CodeLogger utility and check if that is what you want.

davidjryan

@chrishamm @T3P3Tony
Looking at CodeLogger did help me figure a few things, thanks for that push.

After some testing, the EXECUTED mode is what I need. I was only testing the PRE and POST interception modes previously.

I am having issues with the sample program https://github.com/Duet3D/dsf-python/blob/v3.6-dev/examples/custom_m_codes.py

My desire is to capture ALL error messages. Currently I am receiving just those that are generated by G/M/T codes.
I want to do exactly what the first comment in the sample program is stating, capture the whole G-code stream (no filters).

It crashes on the non-G/M/T codes being intercepted (the
cde = intercept_connection.receive_code()
call) and the reported exceptions are a bit Greek to me.

If I change the interception mode from EXECUTED to POST, I get the non-G/M/T intercepts but lose the G/M/T intercepts completely, though the program does not crash. So I don't know if what I am facing is an issue the sample script or the binding call itself.

Who might be the python guru for dealing with the dsf-python bindings? Is it @Falcounet ? I see AndyEveritt is pushing a lot of the contributions for dsf-python on git. Does he have a handle on here?