duetbackup - CLI tool to backup your Duet SD Card [deprecated]

wilriker

This has been deprecated.

Use `rfm` - RepRapFirmware FileManager instead

Original Post Below

I wrote a small command line tool to backup the 0:/sys folder on my Duet regularly. After I mentioned it in another thread and offering binary releases these were requested. So I will publish this a little more so also other people can see and use it if interested.

Description

As stated above it's a command line tool implemented in Go. It knows the following parameters

Usage duetbackup:
    -dirToBackup string
          Directory on Duet to create a backup of (default "0:/sys")
    -domain string
        Domain of Duet Wifi
    -outDir string
          Output dir of backup
    -password string
        Connection password (default "reprap")
    -port uint
          Port of Duet Wifi (default 80)
    -removeLocal
        Remove files locally that have been deleted on the Duet

It will recursively traverse from the start folder down and first load the filelist and then check if a file exists locally and has the same change date. If not it will download the file other it skips this file.

By default it will keep all files locally that have been removed on the SD card but there is a command line flag -removeLocal that will basically synchronize the SD card to the local folder, i.e. also remove files that have been removed remotely.

On the Releases page I provide binaries for Linux, Windows and Mac (darwin) for x86_64 machines as well as two additional binaries for Linux that are targeted towards usage on RaspberryPi (arm and armv7).

If there is any other required binary just let me know and I can easily provide that as well - Go makes this very easy.

Feedback

Any feedback whether it may be bugs or feature requests can be posted here. Also I generally really appreciate Pull Requests.

Example Full SD Backup

If you want to backup all content of your SD card just use

duetbackup -dirToBackup "0:/" -domain <your_domain> -outDir <your_local_backup_dir>

and it will traverse through everything.

gtj0

Getting a "gzip: invalid header" message...

$ ./duetbackup-linux_amd64 -domain duet0 -outDir ./dd/
2019/05/31 12:17:40 Fetching filelist for 0:/sys
2019/05/31 12:17:40 Checking files to be downloaded from 0:/sys
2019/05/31 12:17:40 Adding Duet2CombinedFirmware.bin
2019/05/31 12:17:40 Adding DuetMaestroFirmware.bin
2019/05/31 12:17:40 Adding DuetWebControl.zip
2019/05/31 12:17:40 gzip: invalid header

It did retrieve the 2 bin files but not DuetWebControl.zip

This is a file list of /sys/

ncftp /sys > dir
-rw-rw-rw- 1 ftp ftp          276   May 29 23:59   bed.g
-rw-rw-rw- 1 ftp ftp           79   Aug 15  2018   cancel.g
-rw-rw-rw- 1 ftp ftp         3625   Jan  1  1980   config-2.03.g
-rw-rw-rw- 1 ftp ftp         3811   Dec 27 23:59   config.bak
-rw-rw-rw- 1 ftp ftp         3489   May 30 23:59   config.g
-rw-rw-rw- 1 ftp ftp         3884   Apr 26 23:59   config.g.bak
-rw-rw-rw- 1 ftp ftp         2654   Apr 14  2018   config.json
-rw-rw-rw- 1 ftp ftp          814   May 17 23:59   config-override.g
-rw-rw-rw- 1 ftp ftp         2575   Jul  2  2018   debug.log
-rw-rw-rw- 1 ftp ftp           12   Jul 18  2018   deployprobe.g
-rw-rw-rw- 1 ftp ftp       419120   May 30 23:59   Duet2CombinedFirmware.bin
-rw-rw-rw- 1 ftp ftp       402760   Dec 28 23:59   DuetMaestroFirmware.bin
-rw-rw-rw- 1 ftp ftp       560898   Jan  1  1980   DuetWebControl.zip
-rw-rw-rw- 1 ftp ftp       296832   Dec 28 23:59   DuetWiFiServer.bin
-rw-rw-rw- 1 ftp ftp        63394   May 31  2019   dueui_config_default.json
-rw-rw-rw- 1 ftp ftp        59099   May 31  2019   dueui_config.json
-rw-rw-rw- 1 ftp ftp         1228   Jan 29 23:59   dwc2settings.json
-rw-rw-rw- 1 ftp ftp           97   Jul 13  2018   filaments.csv
-rw-rw-rw- 1 ftp ftp         3119   May 31  2019   heightmap.csv
-rw-rw-rw- 1 ftp ftp          159   Jun 15  2018   homeall.g
-rw-rw-rw- 1 ftp ftp          743   Aug 14  2017   homedelta.g
-rw-rw-rw- 1 ftp ftp           26   Jun 13  2018   homeu.g
-rw-rw-rw- 1 ftp ftp           26   Jun 13  2018   homev.g
-rw-rw-rw- 1 ftp ftp          495   Jun 25  2018   homex.g
-rw-rw-rw- 1 ftp ftp          470   Jun 25  2018   homexy.g
-rw-rw-rw- 1 ftp ftp          495   Jun 25  2018   homey.g
-rw-rw-rw- 1 ftp ftp          392   May 15 23:59   homez.g
-rw-rw-rw- 1 ftp ftp        60476   Dec 28 23:59   iap4e.bin
-rw-rw-rw- 1 ftp ftp        60748   Dec 28 23:59   iap4s.bin
-rw-rw-rw- 1 ftp ftp         1817   Sep 14  2016   LICENSE.txt
-rw-rw-rw- 1 ftp ftp          374   May 10 23:59   pause.g
-rw-rw-rw- 1 ftp ftp          371   Jul 20  2018   resume.g
-rw-rw-rw- 1 ftp ftp          147   Apr 14  2018   sleep.g
-rw-rw-rw- 1 ftp ftp          179   Apr 14  2018   stop.g
-rw-rw-rw- 1 ftp ftp            5   Jun 24  2018   trigger2.g

Oops, meant to add... I'm running 3.0.0beta3

gtj0

Seems like zip files are a problem. Pulling down the "www" directory has the same issue with DuetWebControl.zip but not any other files in "www".

Geez... I forgot to say "Thanks!" for this.
Care to create an uploader as well?

wilriker

@gtj0 Actually you discovered a bug in RRF rather than in my tool.

When RRF sends a ZIP file it will add

Content-Encoding: gzip

to the response header which simply is wrong (except it is a renamed gzip compressed file ). I will though see if I can work around that.

It won't be hard to create an uploader. Will sit down early next week and throw something together.

dc42

There is no reason that I can see for having any .zip files on the SD card. I guess they were either copied to the card manually on a PC, or uploaded some time ago before DWC was clever enough to unzip them and upload the contained files.

wilriker

@dc42 Still if the file is there Content-Encoding: gzip is wrong.

wilriker

Release 1.0.1

I just released v1.0.1 that works around the issue described by @gtj0, i.e. zip files on the SD card provoking an error.

Dougal1957

is the Darwin binary supposed to work on the latest MAC os when I download it it comes as a .dms file which my mac seems to object to? (it seems to think it is a document)

dc42

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

@dc42 Still if the file is there Content-Encoding: gzip is wrong.

I will look into fixing this in the 2.03 release.

Phaedrux

@dougal1957 said in duetbackup - CLI tool to backup your Duet SD Card:

is the Darwin binary supposed to work on the latest MAC os when I download it it comes as a .dms file which my mac seems to object to? (it seems to think it is a document)

Same here.

Dougal1957

@phaedrux Thank God I'm not the only one I am not that good with the intricacies as yet of the Mac and not had much cause to tbh

wilriker

@dougal1957 @Phaedrux yes, the Darwin binary is supposed to work on MacOS 10.10 and later.

I don't know what you tried but for clarification: this is not an installer of any kind. It's the final binary and it's a pure command line tool with no GUI.

So after download open a shell and navigate to the directory the file sits in. Use

chmod +x duetbackup-darwin_amd64

to make it executable and then issue

./duetbackup-darwin_amd64

You should now get an error about missing parameters - but that also means it works.

P.S.: I'm on my phone and the filename was from memory so it might be a tiny bit different, please check that.

Phaedrux

@wilriker that makes more sense. I'll try again tomorrow.

wilriker

@phaedrux let me know if it works this way.

Also I will adjust packaging in the coming days so that a) the filename is the same fur every platform and b) it is already executable for Linux and MacOS. Right now on Windows you would also have to add .exe to the filename to achieve that - I should not do software releases when in a hurry.

mudcruzr

Under windows 10 I run:

.\duetbackup.exe -domain "192.168.0.53" -outDir ".\bkup"

(Tried it with & without the quotes and using ./bkup and .\bkup all with same error.)

and it errors with the following:
2019/06/01 20:15:32 Fetching filelist for 0:/sys
panic: time: missing Location in call to Time.In

The whole error message is here 0_1559416600853_Duetbackup_errors.txt

wilriker

@mudcruzr Thanks for reporting. I'll have a look into that on Monday. This part of the code is rather sketchy and even marked with a // FIXME comment already. So I guess it's time to follow my own advices.

wilriker

Release 1.0.2

I just released version v1.0.2.

This release should fix the time zone issue @mudcruzr described.
Also starting from this release the (callable) filename for all target platforms now is duetbackup - for this the files are package as either tar.gz or zip depending on target

Dougal1957

OK still not sure about this on the mac

I tried the following with the latest 1.02 executable and get the following

0_1559566175798_86beb62e-3019-446f-83b8-ed0598692d8f-image.png

I suspect I need to pass some arguments to it but not sure how? any help appreciated

Doug

wilriker

@dougal1957 Yes, there are mandatory parameters. The names are given in the error message.

To find out all available parameters type

./duetbackup --help

The minimum valid command would look like

./duetbackup -domain "<your_duets_network_address>" -outDir <local_directory_to_save_files>

This will then sync all files in the SD card's /sys (default if -dirToBackup is not provided) directory to the specified directory on your hard drive.

P.S.: For applications implemented in Go it does not matter if you prefix parameters with one or two dashes, i.e. -domain and --domain are treated identically.

mudcruzr

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

Release 1.0.2

I just released version v1.0.2.

This release should fix the time zone issue @mudcruzr described.

Also starting from this release the (callable) filename for all target platforms now is duetbackup - for this the files are package as either tar.gz or zip depending on target

Works for me, many thanks! (Windows 10)

I have a suggestion (only semi-seriously), if it's not too cheeky: an exclude switch, e.g.

./duetbackup -dirToBackup "0:/" -excludeDir "0:/gcodes" -domain "192.168.0.53" -outDir "./bkup"

(I didn't realise how many files I had accumulated in my gcodes directory, lol)

duetbackup - CLI tool to backup your Duet SD Card [deprecated]

This has been deprecated.

Use rfm - RepRapFirmware FileManager instead

Original Post Below

Description

Feedback

Example Full SD Backup

Release 1.0.1

Release 1.0.2

Release 1.0.2

Use `rfm` - RepRapFirmware FileManager instead