Duet3D Logo Duet3D
    • Tags
    • Documentation
    • Order
    • Register
    • Login

    pydsfapi [v3.2.0] - Official Python Client Library for DSF

    Scheduled Pinned Locked Moved
    DSF Development
    6
    51
    3.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • achrnundefined
      achrn @wilriker
      last edited by

      @wilriker
      With the self.socket.recv(32 * 1024) changed to self.socket.recv(64 * 1024) and a routine interrogating the machine model and doing some occasional command_connection.perform_simple_code('M573 P0') at the same time, I've had some prints stop part-way (but neglected to record the error in detail - at the time I thought I knew what had done it so thought I didn't need to pay attention to what it was saying).

      This may be coincidence, though it's the only time the printer has stopped mid-print, but it may be that putting that receive up to 64 kB upsets something (possibly just takes too long).

      wilrikerundefined 1 Reply Last reply Reply Quote 0
      • ofliduetundefined
        ofliduet
        last edited by

        I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

        send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
        recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"
        

        causing a parser error

          File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
            return super().connect(sim, socket_path)
          File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
            response = self.receive_response()
          File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
            return json.loads(json_string, object_hook=responses.decode_response)
          File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
            return cls(**kw).decode(s)
          File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
            raise JSONDecodeError("Extra data", s, end)
        

        Is that a known problem?

        I'm also wondering about whether the API is threadsafe. A complete worked example with a thread running in the background periodically updating a complete machine model with patches would be extremely helpful.

        achrnundefined wilrikerundefined 2 Replies Last reply Reply Quote 0
        • ofliduetundefined
          ofliduet
          last edited by

          Different question: Is it possible to use the API to access the socket from a different host? It looks so close to me but no catch.

          wilrikerundefined 1 Reply Last reply Reply Quote 0
          • achrnundefined
            achrn @ofliduet
            last edited by

            @ofliduet
            Could it be the same issue, but truncating the json in a different location triggering a different error in the json decoder? That is, mine was failing because the truncation happened part-way through a quoted string, if you're not part-way through a string, presumably you'd get a different sort of json decode error?

            One diagnostic would be to print something with very few layers (one or two) to reset the layers dict in the machine model, then try again.

            Also, when I was doing stuff initially I sometimes got an error that if I simply repeated the same script it didn't error. I didn't get to the bottom of that, but one thing I'd try is run the same script again and see if it works.

            ofliduetundefined 1 Reply Last reply Reply Quote 0
            • ofliduetundefined
              ofliduet @achrn
              last edited by

              @achrn Don't think so. This is an initial model, coming in unsolicited directly after the connection has been made. The printer is standing still and I can see the complete model in the recv line. The problem might actually be on the dsf side, where it shouldn't send out a machine model unless specifically requested.

              achrnundefined 1 Reply Last reply Reply Quote 0
              • achrnundefined
                achrn @ofliduet
                last edited by

                @ofliduet
                Standing still doesn't affect it - the machine model seems to contain the last layers list, no matter how long ago that was (though presumably not if the machine and/or the pi is switched off in the meantime, though I haven't tested that).

                However, if you've received the whole machine model, I agree that I don't think it can be the same issue.

                If I do a subscribe connection but don't explicitly request a full machine model, the first 'patch' I get is the full model, but I don't seem to get an unsolicited full model. That is:

                import json
                from pydsfapi import pydsfapi
                from pydsfapi.initmessages.clientinitmessages import SubscriptionMode
                subscribe_connection = pydsfapi.SubscribeConnection(SubscriptionMode.PATCH, debug=False)
                subscribe_connection.connect()
                mm_u_str = subscribe_connection.get_machine_model_patch()
                mm_update=json.loads(mm_u_str)
                

                works fine for me, mm_u_str is a json string of the whole model and json.loads is happy to parse it. Subsequent mm_u_str = subscribe_connection.get_machine_model_patch() give me just small patches, as expected.

                ofliduetundefined 1 Reply Last reply Reply Quote 0
                • ofliduetundefined
                  ofliduet @achrn
                  last edited by

                  @achrn Playing with it a bit more last night I can confirm that the sample code does run correctly when called directly. Where I have problems is when I call the exact some code while there are other things going on.
                  I'm trying to write an Octoprint plugin that provide a virtual printer in Octoprint, so that I can use TheSpaghettiDetective and the Canvas plugins for the Pallette device without having to go through the serial connection. So the code is called up on startup of the Octoprint server when it loads all plugins and the process is obviously busy at that time with multiple threads doing things at the same time. This seems to have an affect, hence my question on the tread safety of the API.

                  1 Reply Last reply Reply Quote 0
                  • wilrikerundefined
                    wilriker @achrn
                    last edited by

                    @achrn said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                    @wilriker
                    With the self.socket.recv(32 * 1024) changed to self.socket.recv(64 * 1024) and a routine interrogating the machine model and doing some occasional command_connection.perform_simple_code('M573 P0') at the same time, I've had some prints stop part-way (but neglected to record the error in detail - at the time I thought I knew what had done it so thought I didn't need to pay attention to what it was saying).

                    If it happens again please record the error and report here.

                    This may be coincidence, though it's the only time the printer has stopped mid-print, but it may be that putting that receive up to 64 kB upsets something (possibly just takes too long).

                    I don't think so. This is all going through a local socket connection, i.e. speeds around RAM speed - 64 KiB will be transferred in under 1ms.

                    Manuel
                    Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                    with probably always latest firmware/DWC (incl. betas or self-compiled)
                    My Tool Collection

                    1 Reply Last reply Reply Quote 0
                    • wilrikerundefined
                      wilriker @ofliduet
                      last edited by

                      @ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                      Different question: Is it possible to use the API to access the socket from a different host? It looks so close to me but no catch.

                      Yes, you can forward the socket via SSH

                      ssh -o StreamLocalBindUnlink=yes -L<localpath>:/run/dsf/dcs.sock <hostname>
                      

                      Make sure that the user you are using to connect via SSH has read-write permissions on /run/dsf/dcs.sock.

                      Manuel
                      Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                      with probably always latest firmware/DWC (incl. betas or self-compiled)
                      My Tool Collection

                      1 Reply Last reply Reply Quote 0
                      • wilrikerundefined
                        wilriker @ofliduet
                        last edited by

                        @ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                        I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

                        send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
                        recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"
                        

                        causing a parser error

                          File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
                            return super().connect(sim, socket_path)
                          File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
                            response = self.receive_response()
                          File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
                            return json.loads(json_string, object_hook=responses.decode_response)
                          File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
                            return cls(**kw).decode(s)
                          File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
                            raise JSONDecodeError("Extra data", s, end)
                        

                        Is that a known problem?

                        No it is not. It looks like DCS was sending only a partial response out of unknown reason. Also it did send this unsolicited as you mentioned in a later post. AFAIK it should not do this.
                        @chrishamm Can you please comment on that part?

                        I'm also wondering about whether the API is threadsafe. A complete worked example with a thread running in the background periodically updating a complete machine model with patches would be extremely helpful.

                        pydsfapi is currently not thread-safe. For the time being you either need to synchronize on the connection yourself or create one connection per thread.

                        Manuel
                        Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                        with probably always latest firmware/DWC (incl. betas or self-compiled)
                        My Tool Collection

                        wilrikerundefined 1 Reply Last reply Reply Quote 0
                        • wilrikerundefined
                          wilriker @achrn
                          last edited by

                          @achrn said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                          Line 284 of pydsfapi/pydsfapi.py limits the json response to 32kB:
                          json_string = self.socket.recv(32 * 1024).decode('utf8')

                          Thanks again for reporting this. I fixed it and will release a new version shortly.

                          Manuel
                          Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                          with probably always latest firmware/DWC (incl. betas or self-compiled)
                          My Tool Collection

                          1 Reply Last reply Reply Quote 0
                          • wilrikerundefined
                            wilriker
                            last edited by

                            Release 3.1.2

                            This is a bugfix release to solve the issue with responses from DCS being capped at 32 KiB.

                            It can be found as usual on GitHub Releases page.

                            Manuel
                            Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                            with probably always latest firmware/DWC (incl. betas or self-compiled)
                            My Tool Collection

                            1 Reply Last reply Reply Quote 0
                            • wilrikerundefined
                              wilriker @wilriker
                              last edited by wilriker

                              @wilriker said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                              @ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

                              I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

                              send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
                              recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"
                              

                              causing a parser error

                                File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
                                  return super().connect(sim, socket_path)
                                File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
                                  response = self.receive_response()
                                File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
                                  return json.loads(json_string, object_hook=responses.decode_response)
                                File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
                                  return cls(**kw).decode(s)
                                File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
                                  raise JSONDecodeError("Extra data", s, end)
                              

                              Is that a known problem?

                              No it is not. It looks like DCS was sending only a partial response out of unknown reason. Also it did send this unsolicited as you mentioned in a later post. AFAIK it should not do this.
                              @chrishamm Can you please comment on that part?

                              My fault. I was confused. This is as intended from DCS and there is actually still a bug in pydsfapi 3.1.2.

                              EDIT: I revise the latter part. AFAICT it should be fixed now.

                              Manuel
                              Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                              with probably always latest firmware/DWC (incl. betas or self-compiled)
                              My Tool Collection

                              achrnundefined 1 Reply Last reply Reply Quote 0
                              • achrnundefined
                                achrn @wilriker
                                last edited by

                                @wilriker Thanks. I've installed and used the new version, though actually my first print job turns out to peak at a 31.9kB machine model, so would have been OK previously (d'oh).

                                However, I do seem to have provoked the error @ofliduet had:

                                Traceback (most recent call last):
                                  File "./mm_time.py", line 12, in <module>
                                    subscribe_connection.connect()
                                  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 447, in connect
                                    return super().connect(sim, socket_path)
                                  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 238, in connect
                                    response = self.receive_response()
                                  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
                                    return json.loads(json_string, object_hook=responses.decode_response)
                                  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
                                    return cls(**kw).decode(s)
                                  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
                                    raise JSONDecodeError("Extra data", s, end)
                                json.decoder.JSONDecodeError: Extra data: line 1 column 17 (char 16)
                                

                                This was from a really simple:

                                #!/usr/bin/python3
                                # subscribe to the machine model and report the timings of most recent print job
                                
                                import json
                                import time
                                from pydsfapi import pydsfapi
                                from pydsfapi.initmessages.clientinitmessages import SubscriptionMode
                                
                                
                                # establish the connection
                                subscribe_connection = pydsfapi.SubscribeConnection(SubscriptionMode.PATCH, debug=False)
                                subscribe_connection.connect()
                                
                                try:
                                    # Get the complete model once
                                    machine_model = subscribe_connection.get_machine_model()
                                
                                    [[continues, just prints out some machine model values, omitted for brevity]]
                                

                                I note that this error is occurring at line 12, before I ask for a copy of the machine model, as @ofliduet reported.

                                If I do literally nothing but run the function again (i.e. press [up-cursor], press [Enter]) it runs fine. About one run in twenty errors, with no discernible pattern. I've never had it error twice in succession, I have had it error, then run OK twice, then error on the next (I think that's the most frequent repeat I've seen).

                                wilrikerundefined 1 Reply Last reply Reply Quote 0
                                • wilrikerundefined
                                  wilriker @achrn
                                  last edited by

                                  @achrn Thanks for re-reporting. I know then what the problem is. I just could not reproduce it on my setup.

                                  I already have a solution that I will provide tomorrow. But I need to find a way to make it more efficient. That will be part of the 3.2 release.

                                  Manuel
                                  Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                                  with probably always latest firmware/DWC (incl. betas or self-compiled)
                                  My Tool Collection

                                  1 Reply Last reply Reply Quote 1
                                  • wilrikerundefined
                                    wilriker
                                    last edited by

                                    Release 3.1.3

                                    This is another bugfix release to solve the problem if message in (primarily in Subscription mode) are received faster than they are processed it would lead to JSON parser errors.

                                    It can be found as usual on GitHub Releases page.

                                    Thanks again to @achrn for reporting.

                                    Manuel
                                    Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                                    with probably always latest firmware/DWC (incl. betas or self-compiled)
                                    My Tool Collection

                                    achrnundefined 1 Reply Last reply Reply Quote 0
                                    • achrnundefined
                                      achrn @wilriker
                                      last edited by

                                      @wilriker Has the type of the MachineModel type that subscribe_conenction.get_machine_model() returns changed?

                                      My scripts that previously did that then e.g. for h in machine_model['heat']['heaters'] are suddenly reporting TypeError: 'MachineModel' object is not subscriptable.

                                      (This might have been a change in 3.1.2 - I had thought I installed 3.1.2, but when I installed 3.1.3 it suggested it was replacing 3.1.1).

                                      wilrikerundefined 2 Replies Last reply Reply Quote 0
                                      • wilrikerundefined
                                        wilriker @achrn
                                        last edited by

                                        @achrn I have noticed something similar but TBH I have no idea what is going on here. Nothing has changed in regards to this but still it is no longer working. I have to investigate that.

                                        Which python version are you using?

                                        Manuel
                                        Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                                        with probably always latest firmware/DWC (incl. betas or self-compiled)
                                        My Tool Collection

                                        achrnundefined 1 Reply Last reply Reply Quote 0
                                        • achrnundefined
                                          achrn @wilriker
                                          last edited by

                                          @wilriker Wow that was quick!

                                          It occurred to me I could check for myself when the change happens - it does change between 3.1.1 and 3.1.2 and if I roll back to 3.1.1 my scripts start working again - MachineModel is subscriptable in 3.1.1, but not in 3.1.2 and 3.1.3.

                                          Having looked at the diff on GitHub 3.1.1 - 3.1.2 I can't see why the changes would change what gets returned (apart from let it be longer). As far as I can see teh changed def receive_json(self): was returning a str and is still returning a str.

                                          My Python is 3.7.3

                                          wilrikerundefined 1 Reply Last reply Reply Quote 0
                                          • wilrikerundefined
                                            wilriker @achrn
                                            last edited by

                                            @achrn Just as an idea, have you removed all .pyc files as well as __pycache__ directories? Just to make sure nothing is stuck in half-way pre-compiled voodoo black-magic bogus.

                                            Manuel
                                            Duet 3 6HC (v0.6) with RPi 4B on a custom Cartesian
                                            with probably always latest firmware/DWC (incl. betas or self-compiled)
                                            My Tool Collection

                                            achrnundefined 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Unless otherwise noted, all forum content is licensed under CC-BY-SA