[3.6.0-beta.3] DWC connection issues Duet2 / Duex5

[timschneider]

@dc42 @chrishamm
thank you both for your time and patience.
I moved forward with the knowledge of the low memory issue and reprogrammed the python client to be a fully fledged digital twin of the duet printer as per object model. In order to archive this I followed the description of M409 and implemented the a and next handling aswell as the seqs handling. So what I do now is, poll the rr_model endpoint with key=seqs to keep track of the sequence numbers and keep the load on the duet as low as possible. If a change happens, it will act accordingly, e.g. fetch a reply or partially fetch the updated object model recursive with max depth of level + 1, again to keep the load as low as possible.

but even that didn't solved the problem - it appears in some cases to make it even worse (i guess two output buffers per json response)

In the following there is one DWC Client and one python client connected.

Anyhow I wanted to understand the problem and at least find a workaround, but I was not able to.

What I found was that on an error - DWC is doing a reconnect straight, which makes it even worse, as this will increase the number of connected clients and will leave gcode replies unreceived and the buffer allocated, I guess?

Using the same macro as above. Directly after sending the command, the following rr_model request in DWC will fail - python client will also receive a HTTP 503 busy and will backoff 5 sec. before even trying to do anything (leaving the session open / timeout=8s - and leaving the reply probably not fetched for this session / due to the 503 busy response). DWC will open a new session and the internal connected clients count is increased, still keeping the buffer allocated. DWC tries to fetch a rr_model and will fail again with HTTP 503 - again reconnect / client count +1 / buffer allocated and so on ...

The last rr_reply in the timeline is empty - and the reconnect and following rr_model request are successfull.

Maybe it is possible to change the behaviour of the DWC reconnect mechanism and check if the HTTP response is 503 - wait and do not reconnect and on the duet side if multiple json requests fail - drop pending gcode replies? Or as an alternative always serve the rr_model seqs requests to give a client a chance to consume rr_reply to free up output buffers. Just randomly polling the rr_reply endpoint will steal the response from DWC, so thats also not an option.

[timschneider]

@timschneider
wow that did the trick, when the python client receives an 503 it will just query the rr_reply immediately until the response is empty to free up buffers on the duet - DWC will not reconnect and everything works even with M98 P"0:/macros/test-bufferoverflow3.g", but DWC will sometimes not receive the output of that macro as the python client consumed both replies.

It will even survive M98 P"0:/macros/test-bufferoverflow4.g"

Using this method having 6 concurrent DWC connections open and one python client is no problem anymore - it will even work with the M98 P"0:/macros/test-bufferoverflow4.g"

Edit: there is still some problem in data and memory management, sometimes i do receive partial data Response payload is not completed: <ContentLengthError: 400, message='Not enough data for satisfy content length header.'> due to connection closing mid transmitting from duet to the reverse proxy.

[chrishamm]

@timschneider What do you have "AJAX retries" in the DWC settings set to?

[timschneider]

@chrishamm