Using pydsfapi with very little knowledge

achrn

I should say at the outset that I have extremely sketchy python knowledge and undertsand even less about most of what I read at https://github.com/Duet3D/DuetSoftwareFramework. Actually, I'm not really comfortable with anything OOP, though I blunder through (the only language I was ever taught was Fortran, I liked Perl before it went OOP). So this might be a struggle...

I have a MB6HC and it's got an attached Pi 4.

I want the Pi to control some stuff in response to what is going on on the Duet, so it seems to me that I want a program running that maintains a copy of the machine model, and can look at its own copy of the machine model and take action accordingly. (An alternative approach would be to periodically do M408 or M409 when my routine wants to know something, but I feel that maintaining a running copy of the machine model is probably less demanding on the Duet.)

As noted, the DSF repository went over my head, but https://github.com/Duet3D/DSF-APIs and particularly pydsfapi lodged slightly more. So from examples.py I can get a program that gets a full copy of the machine model into a dict, then gets sequential updates into a str.

(I spent some time thinking I was supposed to have a json patch, and not understanding why it wasn't working, but after working out that I didn't I realise that the DSF README.md tells me that all along- just as an illustration of how much that went over my head.)

The pydsfapi.py file has a comment "new update patches of the object model need to be applied manually". What I'm wondering is if there's a standard / simple way of doing that which I'm missing? It's not as simple as a dict.update (I think) because the values are themselves dictionaries, some many levels deep. I suspect I can code a recursive function, but I'm feeling I might have missed some standard method or library for doing this.

The other thing I wondered is whether I even need to bother with the whole update / patch mechanism - if my routine is getting the machine model from the DSF that's on the Pi itself, do I even need to worry about the subscription approach, or can I simply regularly grab the complete model safe in the knowledge that it's not going to 'distract' the Duet?

One bonus question, which probably doesn't belong here - can I get the current heater PWM (i.e. what you get from an M573) out of teh machine model? I don't see how.

achrn

Since no-one has leaped in with a clever pythonic way of doing it I've just written (plagiarised) a recursive function. The fact that the machine model is not just a dict of nested dicts but also includes lists of dicts and lists of lists makes it a bit more tricky than I anticipated.

In case it's helpful to anyone else, this is my python function to apply a DSF patch to the machine model. It's the function 'patch', here in a complete example that connects to the DSF and simply prints out the setpoint and current temperature of every defined heater to stdout periodically, but it can obviously be easily modified to do something else - it maintains a copy of the whole machine model.

(But I'm not guaranteeing there isn't a one-line pythonic way of doing this - I might be being clumsy.)

#!/usr/bin/env python3
# subscribe to the machine model and report the temperature each heater

# this is frequency of logging (seconds)
interval=3

# this is duration over which to log (seconds)
duration=90

import json
import sys
import time
from pydsfapi import pydsfapi
from pydsfapi.commands import basecommands, code
from pydsfapi.initmessages.clientinitmessages import InterceptionMode, SubscriptionMode

# function to update dictionary-of-dicts-and-lists a with a merge patch b
# values in b over-write values with same key tree in a
# note this function recurses
# note both a and b are dicts, but pydsfapi patches are just a json string
# so need to be turned into a dict before feeding to this
# built on (i.e. mostly stolen from) http://stackoverflow.com/a/25270947/1431660 
def patch(a, b, path=None, debug=False):
    if path is None: path = []
    if debug and len(path)==0: 
        print("patch the machine model")
        print(b)
    for key in b:
        if debug:
            # debug draws a sort of diagram indicating nesting depth
            print("   ",end="")
            for i in path: print(" > ", end="")
            print(key)
        if key in a:
            # the element in the patch already exists in the model
            if isinstance(a[key], dict) and isinstance(b[key], dict):
                # a[key] and b[key] are both dicts, recurse
                patch(a[key], b[key], path + [str(key)], debug)
            elif isinstance(a[key], list) and isinstance(b[key], list):
                # a[key] and b[key] are both list, enumerate
                for idx, value in enumerate(b[key]):
                    # need new diagram line here for the enumerations 
                    if debug:
                        print("   ",end="")
                        for i in path: print(" > ",end="")
                        # put [] around index to identify we're in a list
                        print(" > ["+str(idx)+"]")
                    # initially assume it's a list of dicts / lists and recurse
                    try:
                        a[key][idx] = patch(a[key][idx], b[key][idx], path + [str(key), str(idx)], debug)
                    except TypeError:
                        # but if that didn't work treat it as a leaf
                        if debug:
                            for i in path: print("   ",end="")
                            print("       = "+str(b[key][idx]))
                        a[key][idx] = b[key][idx]
            else:
                # treat as a leaf, but note either (but not both) could actually be a dict
                # e.g. a[key] could have been a single value and we now overwrite with a new dict
                # or it could have been a dict and now we've overwritten a single scalar
                if debug:
                    for i in path: print("   ",end="")
                    print("    = " + str(b[key]))
                a[key]=b[key]
        else:
            # the key in the path is not yet in the model - just splice it in
            if debug:
                for i in path: print("   ", end="")
                print("*** = " + str(b[key]))
            a[key] = b[key]
    return a

# establish the connection
subscribe_connection = pydsfapi.SubscribeConnection(SubscriptionMode.PATCH, debug=False)
subscribe_connection.connect()

# decide when to stop logging
endat = time.time() + duration

try:
    # Get the complete model once
    machine_model = subscribe_connection.get_machine_model()

    # Get updates
    while time.time() < endat:
        time.sleep(interval - time.time()%interval)
        # get machine model update as a string
        mm_u_str = subscribe_connection.get_machine_model_patch()
        # convert to a  dict
        mm_update=json.loads(mm_u_str)
        # apply to the saved machine model
        patch(machine_model,mm_update)

        # do something with the machine model
        # in this case just print out heater setpoints and current values
        print (time.strftime('%H:%M:%S'), end="")
        for heater in machine_model['heat']['heaters']:
            print (" {:5.1f} {:5.1f}".format(heater['active'],heater['current']), end="")
        print()
finally:
    subscribe_connection.close()

This produces something like:

10:28:12  60.0  60.0   0.0  22.7
10:28:15  60.0  60.0   0.0  22.7
10:28:18  60.0  60.0 230.0  22.6
10:28:21   0.0  60.0 230.0  24.4
10:28:24   0.0  60.0 230.0  28.6
10:28:27   0.0  60.0 230.0  33.5
10:28:30   0.0  59.9 230.0  38.6
10:28:33   0.0  59.9 230.0  43.2
10:28:36   0.0  59.9 230.0  48.7
10:28:39   0.0  59.8 230.0  53.6

Note that the 'patch' function includes a very verbose debug mode (because I wasn't sure what I was doing) which will spew out a sort of diagram of the nesting of the values it is updating:
patch(machine_model,mm_update,debug=True)
gives (for example)

patch machine model
{'boards': [{'mcuTemp': {'current': 39.9}}], 'heat': {'heaters': [{}, {'current': 22.2}]}, 'sensors': {'analog': [{}, {'lastReading': 22.2}, None, None, None, None, None, None, {}, {}]}}
   boards
    > [0]
    >  > mcuTemp
    >  >  > current
             = 39.9
   heat
    > heaters
    >  > [0]
    >  > [1]
    >  >  > current
             = 22.2
   sensors
    > analog
    >  > [0]
    >  > [1]
    >  >  > lastReading
             = 22.2
    >  > [2]
          = None
    >  > [3]
          = None
    >  > [4]
          = None
    >  > [5]
          = None
    >  > [6]
          = None
    >  > [7]
          = None
    >  > [8]
    >  > [9]

One gotcha is that the machine model ['tools'][n]['axes'] is a list of lists and although I think my function handles that OK if it turns up in the patch, I'm pretty sure the debug diagram doesn't show the right thing. That is, I think the machine model might get patched correctly, but the debug diagram is wrong. However, I'm not sure what that part of the machine model is telling me, I haven't seen it change, and I'm not actually that interested in it at the moment, so I haven't set about provoking a change there to test it and fix it.

To answer my question about whether to use the patch mode or just pull the whole model repeatedly I did some very basic checks. I set up two scripts, one doing more-or-less the above example (i.e. getting a patch of changes every three seconds and applying it) and one that pulled the whole model every three seconds. Neither uses enough CPU to show as anything other than 0.0% in top, but running them for some time and ignoring the initial burst of CPU as they initiate, the python time.process_time() is about 50% higher when getting the whole model each time (though that's 50% higher that still registers as 0.0%). So I conclude (with very naive testing) that the patch approach is notionally better, but actually it doesn't matter.

ofliduet

Disclaimer: I'm as new to this as are you (total Python beginner).
That said, I think I have found what you are looking for.

update = subscribe_connection.get_machine_model_patch()
machine_model.__dict__.update(json.loads(update))

ofliduet

Update: There's a caveat. The model isn't always updated correctly when there's an empty entry in an array. The board will return an update like

{"heat":{"heaters":[{},{"current":23}]},"sensors"...

and the empty curly braces will wipe out the model for the first heater. When there is an updated including a new value for heater 0 it's back to normal.

achrn

@ofliduet Thanks for that - I think .update is a standard method on a dict and having done some playing it does obliterate the whole of an item, where the dsf intends simply to be stepping past it with no update.

I've rewritten my 'patch' routine - initially to make the reporting work correctly with lists of lists, but along the way it got much less 'pythonic' (I do type checking in advance of processing variables, which seems to be frowned upon) and much more intelligible. I think I'll post it in a new thread that has a more searchable name, but I'm still testing / checking it at the moment.

MintyTrebor

@achrn : I'm also a python newb, and had a use case similar to yours - needing the pi to control other stuff based on the status/values of the DSF machine model. I created a Python service (using pydsfapi) which allows you to monitor the machine model for value changes and trigger the sending of MQTT msgs to a broker for further action in any system of choice. (I posted about it here).
My code leaves a lot to be desired from a Python point of view, but it may be of some small use to you (I hit similar issues with patches and arrays etc)