GCode file upload with GZIP

resam · 3 Apr 2018, 11:45

Hi,

I'm recently printing more and more larger files (> 50MB). My average upload speed on the DuetWifi is around 500-600 kB/s - resulting in average upload duration for me of 2-3 minutes. What kind of upload speeds are other users achieving? Is my setup alright - or maybe my Wifi is just too slow?

I was wondering if it would make sense to implement a gzip decompressor in RRF, so we can upload a compressed gcode file (temporary file), and RRF then automatically decompresses and writes a new file to the SD card with the actual gcode data.

DWC could transparently gzip-compress the file before uploading. Same for all the slicer integration plugins/scripts. Of course compression should be optional, and make use of the [c]Content-Encoding[/c] HTTP header.

This would also be a good fit for a background task in the new RTOS architecture.
Not sure if the DuetWifi processor is fast enough, or where a potential bottleneck would be (SD card? SPI?).

A quick test on a few of my recent prints indicates achievable compression ratio of up to 5x, e.g. 50MB -> 10MB.

fma · 3 Apr 2018, 13:28

My upload speed is more like 200-300kB/s… But I don't send big files.

I'm wondering if the Duet ethernet can upload at a faster speed?

dc42 · 3 Apr 2018, 13:40

I get 800kbytes/sec to 1Mbyte/sec. Here are some tips:

Use a branded Class 4 or better SD card
Format the SD card with 64K clusters
Don't use a heavily-fragmented SD card
Make sure you have a strong WiFi signal in the vicinty of your printer.

DADIY · 3 Apr 2018, 13:45

@fma:

My upload speed is more like 200-300kB/s… But I don't send big files.

I'm wondering if the Duet ethernet can upload at a faster speed?

I get the same as you with the Duet Ethernet and the SD card supplied with it. I never reformatted the card just used it as is.

dc42 · 3 Apr 2018, 13:49

Duets have been shipping with SD cards formatted to 4kb clusters for a while. That's been changed to 64kb in the latest batch. So it's likely that your card is formatted to 4kb clusters.

I was originally planning to use gzip compression for file uploading when we designed the Duet WiFi, but abandoned the idea when we found we could get good upload speeds without it. But we may look at it again.

fma · 3 Apr 2018, 13:54

Thanks for the tip, David.

resam · 23 Apr 2018, 17:27

Just a little update:
Cura 3.3.0 seems to have implemented some type of compressed G-Code for their UM3 printer:
https://github.com/Ultimaker/Cura/blob/41e3ba528b834f909229f03f4e2e18462de50fae/plugins/ChangeLogPlugin/ChangeLog.txt#L51-L52

Maybe we could make use of this feature with RRF as well!

Phaedrux · 23 Apr 2018, 17:32

I've jumped through the hoops to try and get my upload speeds up. Card has been formatted as FAT16 with 64kb clusters and my printer is right beside the router. Still average around 400kb/s.

Gcode compresses really well, so it seems like low hanging fruit.

deckingman · 23 Apr 2018, 19:32

If it's any help, I use SanDisc Extreme (not the Ultra) SD cards. Usually 32 GB. Can't say what the cluster size is as I don't reformat them - use them just as they come. Like David, I get 800 to 1,000 Kbytes/sec. Mine is Ethernet rather than WiFi if that makes any difference.

Guyaros · 2 May 2018, 20:43

@dc42 said in GCode file upload with GZIP:

I get 800kbytes/sec to 1Mbyte/sec. Here are some tips:

Use a branded Class 4 or better SD card

Format the SD card with 64K clusters

Is it ok to format the SD card using windows 10 format tool?

The memory card is Sandisk Ultra Class 10 16GB

dc42 · 2 May 2018, 20:47

Yes, you can use the Windows formatting tool.