Possible to send binary data with M559 over Telnet?

Otso

I'm continuing with my desktop app that communicates over Telnet. I'm now implementing file upload and using M559 P"filename" Snnn to upload files as binary. I'm not able to get it to work as expected. I must send much more data than provided in the S parameter before the Duet 2 Wifi thinks it has received all bytes. I tried sending just a few bytes at a time at a slow pace, but that didn't help. A quick look at the RepRapFirmware source code seems to indicate it always reads a line at time.

For example, if I send the same 100 byte file, but one ending in a newline char, and the other not (and instead ending in, say, the character 'A'), then only the file ending in the newline char will be uploaded. So it seems to be relying on processing a line at time and gets mixed up in the byte counting.

Is M559 P"filename" Snnn over Telnet supposed to work (with arbitrary binary content)?

Otso

@otso I'm pretty sure this is a bug. How to repeat:

Create file1.txt with contents "012345678\n" (file size 10 bytes)
Create file2.txt with contents "0123456789" (file size 10 bytes)
Send M559 P"file1.txt" S10
Transmit the 10 bytes of file1.txt
File is successfully uploaded
Send M559 P"file2.txt" S10
Transmit the 10 bytes of file2.txt
The Duet2 with RepRapFirmware 3 waits for more input
You have to transmit more data after this, until it is accepted. A single newline is not enough.

I have verified that the 10 bytes are actually transmitted and acknowledged on the TCP level by dumping the communication with tcpdump. So it's not that the data is buffered by the transmitter.

Some more testing:

Transmit "0123456789" (10 bytes, no newline). All 10 bytes are sent in a single TCP packet, and acknowledged by the Duet.
Duet2 waits for more data
Transmit "54321\n" (6 bytes in a single TCP packet)
Duet wait for more data
Transmit 54321\n" (6 bytes in a single TCP packet)
File is saved. (and "1" is interpreted as a command)
Contents of the file is "54321\n5432" (10 bytes)

So the first 10 bytes are completely discarded, and only the 10 following bytes are saved.

Phaedrux

Firmware version?

Otso

@phaedrux FIRMWARE_NAME: RepRapFirmware for Duet 2 WiFi/Ethernet FIRMWARE_VERSION: 3.3 ELECTRONICS: Duet WiFi 1.02 or later FIRMWARE_DATE: 2021-06-15 21:45:03

Phaedrux

Can I ask that you test with the 3.4 final firmware release?

Otso

@phaedrux Sure. I'll have to read in a bit on how to do it first, since previously I've just followed the instructions provided by Ooznest without actually knowing what I've been doing. (https://learn.ooznest.co.uk/Guide/How+To+Update+WorkBee+Firmware+V1.0.7+>+V3.3.0-1.2/240)

Phaedrux

It's as easy as uploading the zip file for the 3.4 release to the system tab in DWC.

https://github.com/Duet3D/RepRapFirmware/releases/download/3.4.0/Duet2and3Firmware-3.4.0.zip

Don't extract it first. It will take care of the rest.

Otso

@phaedrux Thanks! I was just reading that, but slightly hesitating since Ooznest has made a customized DWC for the WorkBee CNC (optimized for CNC operation rather than 3D printing). I guess I can just overwrite DWC later with the Ooznest version? Not that it is that important anymore since I now mainly use my own desktop app anyway, and not DWC. (Currently I mainly need DWC to upload my G-code files )

Edit: The DWC differences between the default and Ooznest version is minor, smaller than I remembered. It makes no practical difference, so there's no need for me to replace the default standard version with the Ooznest version.

Otso

@otso Now upgraded to 3.4 and the issue persists.

Otso

@otso But I might as well implement the file upload using HTTP, since I realized there isn't any way to download files over Telnet, right? (For example if you want to edit files). So I might do both file upload and download over HTTP, and then M559 will not be an issue for me.

Phaedrux

I can't imagine telnet being the preferred method for anything these days.

Otso

@phaedrux I prefer Telnet in this case as it keeps the connection open and there is no overhead. Websockets would be an improvement (but still has unnecessary overhead). The "raw" connection of Telnet is nice and clean, and my app will work the same if connected over USB or serial.

Otso

@Phaedrux I'm running in Marlin compatibility mode, if that makes a difference for the M559 issue.

Edit: Does not make a difference if I switch to RepRapFirmware mode.

Phaedrux

I'll see if DC42 can take a look at your telnet question.

Otso

@phaedrux Thank you! Please keep it at as a low priority, unless you have a general interest in fixing it. It's not that important or critical for me. A note in the documentation would be helpful, however.

Otso

@phaedrux With a quick test, sending with M559 without the Snnn parameter and instead terminating with M29 doesn't seem to work either:

M559 P"/gcodes/foobar.txt"
Writing to file: /gcodes/foobar.txt
ok
Hello
World
M29

I would expect ok after M29. The only way I can continue is rebooting.

OwenD

M559 is supposed to be terminated by sending M29
I don't see that in your original test.
On the one where it froze did you send a line feed before M29?
Perhaps try again with the S parameter and M29

You might also try M560
It has a special EOF line

chrishamm

@otso The Telnet responder isn't designed to deal with binary file uploads. The reason is that Telnet data is processed line by line and if the line length exceeds the maximum length (255 chars), the line content is dropped and an error message is written to the USB line (if possible).

So, at least for now, you should use the rr_upload HTTP request for binary data. I'll discuss with @dc42 how we will treat M559 from Telnet in the next RRF version.