Duet3D Logo Duet3D
    • Tags
    • Documentation
    • Order
    • Register
    • Login

    pydsfapi [v3.2.0] - Official Python Client Library for DSF

    Scheduled Pinned Locked Moved
    DSF Development
    6
    51
    3.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • achrnundefined
      achrn @ofliduet
      last edited by

      @ofliduet
      Standing still doesn't affect it - the machine model seems to contain the last layers list, no matter how long ago that was (though presumably not if the machine and/or the pi is switched off in the meantime, though I haven't tested that).

      However, if you've received the whole machine model, I agree that I don't think it can be the same issue.

      If I do a subscribe connection but don't explicitly request a full machine model, the first 'patch' I get is the full model, but I don't seem to get an unsolicited full model. That is:

      import json
      from pydsfapi import pydsfapi
      from pydsfapi.initmessages.clientinitmessages import SubscriptionMode
      subscribe_connection = pydsfapi.SubscribeConnection(SubscriptionMode.PATCH, debug=False)
      subscribe_connection.connect()
      mm_u_str = subscribe_connection.get_machine_model_patch()
      mm_update=json.loads(mm_u_str)
      

      works fine for me, mm_u_str is a json string of the whole model and json.loads is happy to parse it. Subsequent mm_u_str = subscribe_connection.get_machine_model_patch() give me just small patches, as expected.

      ofliduetundefined 1 Reply Last reply Reply Quote 0
      • ofliduetundefined
        ofliduet @achrn
        last edited by

        @achrn Playing with it a bit more last night I can confirm that the sample code does run correctly when called directly. Where I have problems is when I call the exact some code while there are other things going on.
        I'm trying to write an Octoprint plugin that provide a virtual printer in Octoprint, so that I can use TheSpaghettiDetective and the Canvas plugins for the Pallette device without having to go through the serial connection. So the code is called up on startup of the Octoprint server when it loads all plugins and the process is obviously busy at that time with multiple threads doing things at the same time. This seems to have an affect, hence my question on the tread safety of the API.

        1 Reply Last reply Reply Quote 0
        • wilrikerundefined
          wilriker @achrn
          last edited by

          @achrn said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

          @wilriker
          With the self.socket.recv(32 * 1024) changed to self.socket.recv(64 * 1024) and a routine interrogating the machine model and doing some occasional command_connection.perform_simple_code('M573 P0') at the same time, I've had some prints stop part-way (but neglected to record the error in detail - at the time I thought I knew what had done it so thought I didn't need to pay attention to what it was saying).

          If it happens again please record the error and report here.

          This may be coincidence, though it's the only time the printer has stopped mid-print, but it may be that putting that receive up to 64 kB upsets something (possibly just takes too long).

          I don't think so. This is all going through a local socket connection, i.e. speeds around RAM speed - 64 KiB will be transferred in under 1ms.

          Manuel
          Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
          with probably always latest firmware/DWC (incl. betas or self-compiled)
          My Tool Collection

          1 Reply Last reply Reply Quote 0
          • wilrikerundefined
            wilriker @ofliduet
            last edited by

            @ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

            Different question: Is it possible to use the API to access the socket from a different host? It looks so close to me but no catch.

            Yes, you can forward the socket via SSH

            ssh -o StreamLocalBindUnlink=yes -L<localpath>:/run/dsf/dcs.sock <hostname>
            

            Make sure that the user you are using to connect via SSH has read-write permissions on /run/dsf/dcs.sock.

            Manuel
            Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
            with probably always latest firmware/DWC (incl. betas or self-compiled)
            My Tool Collection

            1 Reply Last reply Reply Quote 0
            • wilrikerundefined
              wilriker @ofliduet
              last edited by

              @ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

              I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

              send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
              recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"
              

              causing a parser error

                File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
                  return super().connect(sim, socket_path)
                File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
                  response = self.receive_response()
                File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
                  return json.loads(json_string, object_hook=responses.decode_response)
                File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
                  return cls(**kw).decode(s)
                File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
                  raise JSONDecodeError("Extra data", s, end)
              

              Is that a known problem?

              No it is not. It looks like DCS was sending only a partial response out of unknown reason. Also it did send this unsolicited as you mentioned in a later post. AFAIK it should not do this.
              @chrishamm Can you please comment on that part?

              I'm also wondering about whether the API is threadsafe. A complete worked example with a thread running in the background periodically updating a complete machine model with patches would be extremely helpful.

              pydsfapi is currently not thread-safe. For the time being you either need to synchronize on the connection yourself or create one connection per thread.

              Manuel
              Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
              with probably always latest firmware/DWC (incl. betas or self-compiled)
              My Tool Collection

              wilrikerundefined 1 Reply Last reply Reply Quote 0
              • wilrikerundefined
                wilriker @achrn
                last edited by

                @achrn said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                Line 284 of pydsfapi/pydsfapi.py limits the json response to 32kB:
                json_string = self.socket.recv(32 * 1024).decode('utf8')

                Thanks again for reporting this. I fixed it and will release a new version shortly.

                Manuel
                Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                with probably always latest firmware/DWC (incl. betas or self-compiled)
                My Tool Collection

                1 Reply Last reply Reply Quote 0
                • wilrikerundefined
                  wilriker
                  last edited by

                  Release 3.1.2

                  This is a bugfix release to solve the issue with responses from DCS being capped at 32 KiB.

                  It can be found as usual on GitHub Releases page.

                  Manuel
                  Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                  with probably always latest firmware/DWC (incl. betas or self-compiled)
                  My Tool Collection

                  1 Reply Last reply Reply Quote 0
                  • wilrikerundefined
                    wilriker @wilriker
                    last edited by wilriker

                    @wilriker said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                    @ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                    I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

                    send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
                    recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"
                    

                    causing a parser error

                      File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
                        return super().connect(sim, socket_path)
                      File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
                        response = self.receive_response()
                      File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
                        return json.loads(json_string, object_hook=responses.decode_response)
                      File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
                        return cls(**kw).decode(s)
                      File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
                        raise JSONDecodeError("Extra data", s, end)
                    

                    Is that a known problem?

                    No it is not. It looks like DCS was sending only a partial response out of unknown reason. Also it did send this unsolicited as you mentioned in a later post. AFAIK it should not do this.
                    @chrishamm Can you please comment on that part?

                    My fault. I was confused. This is as intended from DCS and there is actually still a bug in pydsfapi 3.1.2.

                    EDIT: I revise the latter part. AFAICT it should be fixed now.

                    Manuel
                    Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                    with probably always latest firmware/DWC (incl. betas or self-compiled)
                    My Tool Collection

                    achrnundefined 1 Reply Last reply Reply Quote 0
                    • achrnundefined
                      achrn @wilriker
                      last edited by

                      @wilriker Thanks. I've installed and used the new version, though actually my first print job turns out to peak at a 31.9kB machine model, so would have been OK previously (d'oh).

                      However, I do seem to have provoked the error @ofliduet had:

                      Traceback (most recent call last):
                        File "./mm_time.py", line 12, in <module>
                          subscribe_connection.connect()
                        File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 447, in connect
                          return super().connect(sim, socket_path)
                        File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 238, in connect
                          response = self.receive_response()
                        File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
                          return json.loads(json_string, object_hook=responses.decode_response)
                        File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
                          return cls(**kw).decode(s)
                        File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
                          raise JSONDecodeError("Extra data", s, end)
                      json.decoder.JSONDecodeError: Extra data: line 1 column 17 (char 16)
                      

                      This was from a really simple:

                      #!/usr/bin/python3
                      # subscribe to the machine model and report the timings of most recent print job
                      
                      import json
                      import time
                      from pydsfapi import pydsfapi
                      from pydsfapi.initmessages.clientinitmessages import SubscriptionMode
                      
                      
                      # establish the connection
                      subscribe_connection = pydsfapi.SubscribeConnection(SubscriptionMode.PATCH, debug=False)
                      subscribe_connection.connect()
                      
                      try:
                          # Get the complete model once
                          machine_model = subscribe_connection.get_machine_model()
                      
                          [[continues, just prints out some machine model values, omitted for brevity]]
                      

                      I note that this error is occurring at line 12, before I ask for a copy of the machine model, as @ofliduet reported.

                      If I do literally nothing but run the function again (i.e. press [up-cursor], press [Enter]) it runs fine. About one run in twenty errors, with no discernible pattern. I've never had it error twice in succession, I have had it error, then run OK twice, then error on the next (I think that's the most frequent repeat I've seen).

                      wilrikerundefined 1 Reply Last reply Reply Quote 0
                      • wilrikerundefined
                        wilriker @achrn
                        last edited by

                        @achrn Thanks for re-reporting. I know then what the problem is. I just could not reproduce it on my setup.

                        I already have a solution that I will provide tomorrow. But I need to find a way to make it more efficient. That will be part of the 3.2 release.

                        Manuel
                        Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                        with probably always latest firmware/DWC (incl. betas or self-compiled)
                        My Tool Collection

                        1 Reply Last reply Reply Quote 1
                        • wilrikerundefined
                          wilriker
                          last edited by

                          Release 3.1.3

                          This is another bugfix release to solve the problem if message in (primarily in Subscription mode) are received faster than they are processed it would lead to JSON parser errors.

                          It can be found as usual on GitHub Releases page.

                          Thanks again to @achrn for reporting.

                          Manuel
                          Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                          with probably always latest firmware/DWC (incl. betas or self-compiled)
                          My Tool Collection

                          achrnundefined 1 Reply Last reply Reply Quote 0
                          • achrnundefined
                            achrn @wilriker
                            last edited by

                            @wilriker Has the type of the MachineModel type that subscribe_conenction.get_machine_model() returns changed?

                            My scripts that previously did that then e.g. for h in machine_model['heat']['heaters'] are suddenly reporting TypeError: 'MachineModel' object is not subscriptable.

                            (This might have been a change in 3.1.2 - I had thought I installed 3.1.2, but when I installed 3.1.3 it suggested it was replacing 3.1.1).

                            wilrikerundefined 2 Replies Last reply Reply Quote 0
                            • wilrikerundefined
                              wilriker @achrn
                              last edited by

                              @achrn I have noticed something similar but TBH I have no idea what is going on here. Nothing has changed in regards to this but still it is no longer working. I have to investigate that.

                              Which python version are you using?

                              Manuel
                              Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                              with probably always latest firmware/DWC (incl. betas or self-compiled)
                              My Tool Collection

                              achrnundefined 1 Reply Last reply Reply Quote 0
                              • achrnundefined
                                achrn @wilriker
                                last edited by

                                @wilriker Wow that was quick!

                                It occurred to me I could check for myself when the change happens - it does change between 3.1.1 and 3.1.2 and if I roll back to 3.1.1 my scripts start working again - MachineModel is subscriptable in 3.1.1, but not in 3.1.2 and 3.1.3.

                                Having looked at the diff on GitHub 3.1.1 - 3.1.2 I can't see why the changes would change what gets returned (apart from let it be longer). As far as I can see teh changed def receive_json(self): was returning a str and is still returning a str.

                                My Python is 3.7.3

                                wilrikerundefined 1 Reply Last reply Reply Quote 0
                                • wilrikerundefined
                                  wilriker @achrn
                                  last edited by

                                  @achrn Just as an idea, have you removed all .pyc files as well as __pycache__ directories? Just to make sure nothing is stuck in half-way pre-compiled voodoo black-magic bogus.

                                  Manuel
                                  Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                                  with probably always latest firmware/DWC (incl. betas or self-compiled)
                                  My Tool Collection

                                  achrnundefined 1 Reply Last reply Reply Quote 0
                                  • achrnundefined
                                    achrn @wilriker
                                    last edited by

                                    @wilriker I ahdan't, but I was doing pip3 uninstall pydsfapi before each new install.

                                    I've tried again - unistall, hunt for any pydsfapi traces and obliterate them (including fo ~/.local/lib/python3.7 where I found a few). I'm not sure I want to remove all __pycache__ and *.pyc - plus it would take a while to hunt them all down. There are no *dsf*.pyc or pydsfapi* (except the .gz I'm installing from) files anywhere on the machine before I do a pip3 install

                                    After that, still the same behaviour - MachineModel object is not subscriptable.

                                    I have a relatively simple workaround, because if I don't transfer a full model, my first 'patch' gets a whole model. So I can easily change my scripts to start with an empty dict, and use get_machine_model_patch() to obtain a 'patch' containing the whole model (as a nice predictable string). Then I apply that patch to the empty dict, and that gives me a dict of the machine model, and the script proceeds from that point fine.

                                    wilrikerundefined 1 Reply Last reply Reply Quote 0
                                    • wilrikerundefined
                                      wilriker @achrn
                                      last edited by

                                      @achrn I have a vague theory of what might be the cause for this. But I can probably only test next week.

                                      Manuel
                                      Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                                      with probably always latest firmware/DWC (incl. betas or self-compiled)
                                      My Tool Collection

                                      achrnundefined 1 Reply Last reply Reply Quote 0
                                      • achrnundefined
                                        achrn @wilriker
                                        last edited by

                                        @wilriker No problem - as I say, I've got an easy workaround, so there's not a rush on my account. Thanks.

                                        1 Reply Last reply Reply Quote 0
                                        • wilrikerundefined
                                          wilriker @achrn
                                          last edited by

                                          @achrn said in pydsfapi [v3.1.3] - Official Python Client Library for DSF:

                                          My scripts that previously did that then e.g. for h in machine_model['heat']['heaters'] are suddenly reporting TypeError: 'MachineModel' object is not subscriptable.

                                          I see the same problem with 3.1.1. It works for me (both versions) if I use

                                          for h in machine_model.heat['heaters']
                                          

                                          i.e. trying to access heat as a member rather than a dict key.

                                          Manuel
                                          Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                                          with probably always latest firmware/DWC (incl. betas or self-compiled)
                                          My Tool Collection

                                          achrnundefined 1 Reply Last reply Reply Quote 0
                                          • achrnundefined
                                            achrn @wilriker
                                            last edited by

                                            @wilriker said in pydsfapi [v3.1.3] - Official Python Client Library for DSF:

                                            It works for me (both versions) if I use
                                            for h in machine_model.heat['heaters']

                                            Yes, that works for me too in 3.1.3.

                                            However, it breaks my patch-applying routine, so I think it's more work for me than my other workaround (at least, for my existing scripts).

                                            wilrikerundefined 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Unless otherwise noted, all forum content is licensed under CC-BY-SA