pydsfapi [v3.2.0] - Official Python Client Library for DSF

achrn

Line 284 of pydsfapi/pydsfapi.py limits the json response to 32kB:
json_string = self.socket.recv(32 * 1024).decode('utf8')

The full machine model can easily exceed that when a print is under way, because it contains an entry for every layer, with at least 90 bytes per layer (depending on the values and number of filaments). Then the json string is just arbitrarily truncated at 32kB and you get an error such as:

Traceback (most recent call last):
  File "./delta_lights.py", line 99, in <module>
    mm = subscribe_connection.get_machine_model()
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 455, in get_machine_model
    machine_model = self.receive(machinemodel.MachineModel)
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 275, in receive
    return cls.from_json(json.loads(json_string))
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 32768 (char 32767)

If you change the (32 * 1024) to something bigger it goes away, but it doesn't take a very large print to overwhelm any sensible value - 32kB at 90 bytes per layer is only 364 layers, so another 32kB gets you less than a further 73mm of print height at 0.2mm layers.

As I understand it, there's the facility in the DSF to filter the machine model returned. However, I haven't figured out of that is in pydsfapi and if so how to use it. Can a filter be used to not return the 'layers' entries?

wilriker

@achrn Thanks for reporting. I will look into that.

achrn

@wilriker
With the self.socket.recv(32 * 1024) changed to self.socket.recv(64 * 1024) and a routine interrogating the machine model and doing some occasional command_connection.perform_simple_code('M573 P0') at the same time, I've had some prints stop part-way (but neglected to record the error in detail - at the time I thought I knew what had done it so thought I didn't need to pay attention to what it was saying).

This may be coincidence, though it's the only time the printer has stopped mid-print, but it may be that putting that receive up to 64 kB upsets something (possibly just takes too long).

ofliduet

I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"

causing a parser error

  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
    return super().connect(sim, socket_path)
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)

Is that a known problem?

I'm also wondering about whether the API is threadsafe. A complete worked example with a thread running in the background periodically updating a complete machine model with patches would be extremely helpful.

ofliduet

Different question: Is it possible to use the API to access the socket from a different host? It looks so close to me but no catch.

achrn

@ofliduet
Could it be the same issue, but truncating the json in a different location triggering a different error in the json decoder? That is, mine was failing because the truncation happened part-way through a quoted string, if you're not part-way through a string, presumably you'd get a different sort of json decode error?

One diagnostic would be to print something with very few layers (one or two) to reset the layers dict in the machine model, then try again.

Also, when I was doing stuff initially I sometimes got an error that if I simply repeated the same script it didn't error. I didn't get to the bottom of that, but one thing I'd try is run the same script again and see if it works.

ofliduet

@achrn Don't think so. This is an initial model, coming in unsolicited directly after the connection has been made. The printer is standing still and I can see the complete model in the recv line. The problem might actually be on the dsf side, where it shouldn't send out a machine model unless specifically requested.

achrn

@ofliduet
Standing still doesn't affect it - the machine model seems to contain the last layers list, no matter how long ago that was (though presumably not if the machine and/or the pi is switched off in the meantime, though I haven't tested that).

However, if you've received the whole machine model, I agree that I don't think it can be the same issue.

If I do a subscribe connection but don't explicitly request a full machine model, the first 'patch' I get is the full model, but I don't seem to get an unsolicited full model. That is:

import json
from pydsfapi import pydsfapi
from pydsfapi.initmessages.clientinitmessages import SubscriptionMode
subscribe_connection = pydsfapi.SubscribeConnection(SubscriptionMode.PATCH, debug=False)
subscribe_connection.connect()
mm_u_str = subscribe_connection.get_machine_model_patch()
mm_update=json.loads(mm_u_str)

works fine for me, mm_u_str is a json string of the whole model and json.loads is happy to parse it. Subsequent mm_u_str = subscribe_connection.get_machine_model_patch() give me just small patches, as expected.

ofliduet

@achrn Playing with it a bit more last night I can confirm that the sample code does run correctly when called directly. Where I have problems is when I call the exact some code while there are other things going on.
I'm trying to write an Octoprint plugin that provide a virtual printer in Octoprint, so that I can use TheSpaghettiDetective and the Canvas plugins for the Pallette device without having to go through the serial connection. So the code is called up on startup of the Octoprint server when it loads all plugins and the process is obviously busy at that time with multiple threads doing things at the same time. This seems to have an affect, hence my question on the tread safety of the API.

wilriker

@achrn said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

@wilriker
With the self.socket.recv(32 * 1024) changed to self.socket.recv(64 * 1024) and a routine interrogating the machine model and doing some occasional command_connection.perform_simple_code('M573 P0') at the same time, I've had some prints stop part-way (but neglected to record the error in detail - at the time I thought I knew what had done it so thought I didn't need to pay attention to what it was saying).

If it happens again please record the error and report here.

This may be coincidence, though it's the only time the printer has stopped mid-print, but it may be that putting that receive up to 64 kB upsets something (possibly just takes too long).

I don't think so. This is all going through a local socket connection, i.e. speeds around RAM speed - 64 KiB will be transferred in under 1ms.

wilriker

@ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

Different question: Is it possible to use the API to access the socket from a different host? It looks so close to me but no catch.

Yes, you can forward the socket via SSH

ssh -o StreamLocalBindUnlink=yes -L<localpath>:/run/dsf/dcs.sock <hostname>

Make sure that the user you are using to connect via SSH has read-write permissions on /run/dsf/dcs.sock.

wilriker

@ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"

causing a parser error

  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
    return super().connect(sim, socket_path)
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)

Is that a known problem?

No it is not. It looks like DCS was sending only a partial response out of unknown reason. Also it did send this unsolicited as you mentioned in a later post. AFAIK it should not do this.
@chrishamm Can you please comment on that part?

I'm also wondering about whether the API is threadsafe. A complete worked example with a thread running in the background periodically updating a complete machine model with patches would be extremely helpful.

pydsfapi is currently not thread-safe. For the time being you either need to synchronize on the connection yourself or create one connection per thread.

wilriker

@achrn said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

Line 284 of pydsfapi/pydsfapi.py limits the json response to 32kB:
json_string = self.socket.recv(32 * 1024).decode('utf8')

Thanks again for reporting this. I fixed it and will release a new version shortly.

wilriker

Release 3.1.2

This is a bugfix release to solve the issue with responses from DCS being capped at 32 KiB.

It can be found as usual on GitHub Releases page.

wilriker

@wilriker said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

@ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:
I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:
send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"
causing a parser error
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
    return super().connect(sim, socket_path)
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
Is that a known problem?
No it is not. It looks like DCS was sending only a partial response out of unknown reason. Also it did send this unsolicited as you mentioned in a later post. AFAIK it should not do this.
@chrishamm Can you please comment on that part?

My fault. I was confused. This is as intended from DCS ~~and there is actually still a bug in pydsfapi 3.1.2.~~

EDIT: I revise the latter part. AFAICT it should be fixed now.

achrn

@wilriker Thanks. I've installed and used the new version, though actually my first print job turns out to peak at a 31.9kB machine model, so would have been OK previously (d'oh).

However, I do seem to have provoked the error @ofliduet had:

Traceback (most recent call last):
  File "./mm_time.py", line 12, in <module>
    subscribe_connection.connect()
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 447, in connect
    return super().connect(sim, socket_path)
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 17 (char 16)

This was from a really simple:

#!/usr/bin/python3
# subscribe to the machine model and report the timings of most recent print job

import json
import time
from pydsfapi import pydsfapi
from pydsfapi.initmessages.clientinitmessages import SubscriptionMode


# establish the connection
subscribe_connection = pydsfapi.SubscribeConnection(SubscriptionMode.PATCH, debug=False)
subscribe_connection.connect()

try:
    # Get the complete model once
    machine_model = subscribe_connection.get_machine_model()

    [[continues, just prints out some machine model values, omitted for brevity]]

I note that this error is occurring at line 12, before I ask for a copy of the machine model, as @ofliduet reported.

If I do literally nothing but run the function again (i.e. press [up-cursor], press [Enter]) it runs fine. About one run in twenty errors, with no discernible pattern. I've never had it error twice in succession, I have had it error, then run OK twice, then error on the next (I think that's the most frequent repeat I've seen).

wilriker

@achrn Thanks for re-reporting. I know then what the problem is. I just could not reproduce it on my setup.

I already have a solution that I will provide tomorrow. But I need to find a way to make it more efficient. That will be part of the 3.2 release.

wilriker

Release 3.1.3

This is another bugfix release to solve the problem if message in (primarily in Subscription mode) are received faster than they are processed it would lead to JSON parser errors.

It can be found as usual on GitHub Releases page.

Thanks again to @achrn for reporting.

achrn

@wilriker Has the type of the MachineModel type that subscribe_conenction.get_machine_model() returns changed?

My scripts that previously did that then e.g. for h in machine_model['heat']['heaters'] are suddenly reporting TypeError: 'MachineModel' object is not subscriptable.

(This might have been a change in 3.1.2 - I had thought I installed 3.1.2, but when I installed 3.1.3 it suggested it was replacing 3.1.1).

wilriker

@achrn I have noticed something similar but TBH I have no idea what is going on here. Nothing has changed in regards to this but still it is no longer working. I have to investigate that.

Which python version are you using?