duetbackup - CLI tool to backup your Duet SD Card [deprecated]



  • This has been deprecated.

    Use rfm - RepRapFirmware FileManager instead

    Original Post Below

    I wrote a small command line tool to backup the 0:/sys folder on my Duet regularly. After I mentioned it in another thread and offering binary releases these were requested. So I will publish this a little more so also other people can see and use it if interested.

    Description

    As stated above it's a command line tool implemented in Go. It knows the following parameters

    Usage duetbackup:
        -dirToBackup string
              Directory on Duet to create a backup of (default "0:/sys")
        -domain string
            Domain of Duet Wifi
        -outDir string
              Output dir of backup
        -password string
            Connection password (default "reprap")
        -port uint
              Port of Duet Wifi (default 80)
        -removeLocal
            Remove files locally that have been deleted on the Duet
    

    It will recursively traverse from the start folder down and first load the filelist and then check if a file exists locally and has the same change date. If not it will download the file other it skips this file.

    By default it will keep all files locally that have been removed on the SD card but there is a command line flag -removeLocal that will basically synchronize the SD card to the local folder, i.e. also remove files that have been removed remotely.

    On the Releases page I provide binaries for Linux, Windows and Mac (darwin) for x86_64 machines as well as two additional binaries for Linux that are targeted towards usage on RaspberryPi (arm and armv7).

    If there is any other required binary just let me know and I can easily provide that as well - Go makes this very easy.

    Feedback

    Any feedback whether it may be bugs or feature requests can be posted here. Also I generally really appreciate Pull Requests. 🙂

    Example Full SD Backup

    If you want to backup all content of your SD card just use

    duetbackup -dirToBackup "0:/" -domain <your_domain> -outDir <your_local_backup_dir>
    

    and it will traverse through everything.



  • Getting a "gzip: invalid header" message...

    $ ./duetbackup-linux_amd64 -domain duet0 -outDir ./dd/
    2019/05/31 12:17:40 Fetching filelist for 0:/sys
    2019/05/31 12:17:40 Checking files to be downloaded from 0:/sys
    2019/05/31 12:17:40 Adding Duet2CombinedFirmware.bin
    2019/05/31 12:17:40 Adding DuetMaestroFirmware.bin
    2019/05/31 12:17:40 Adding DuetWebControl.zip
    2019/05/31 12:17:40 gzip: invalid header
    

    It did retrieve the 2 bin files but not DuetWebControl.zip

    This is a file list of /sys/

    ncftp /sys > dir
    -rw-rw-rw- 1 ftp ftp          276   May 29 23:59   bed.g
    -rw-rw-rw- 1 ftp ftp           79   Aug 15  2018   cancel.g
    -rw-rw-rw- 1 ftp ftp         3625   Jan  1  1980   config-2.03.g
    -rw-rw-rw- 1 ftp ftp         3811   Dec 27 23:59   config.bak
    -rw-rw-rw- 1 ftp ftp         3489   May 30 23:59   config.g
    -rw-rw-rw- 1 ftp ftp         3884   Apr 26 23:59   config.g.bak
    -rw-rw-rw- 1 ftp ftp         2654   Apr 14  2018   config.json
    -rw-rw-rw- 1 ftp ftp          814   May 17 23:59   config-override.g
    -rw-rw-rw- 1 ftp ftp         2575   Jul  2  2018   debug.log
    -rw-rw-rw- 1 ftp ftp           12   Jul 18  2018   deployprobe.g
    -rw-rw-rw- 1 ftp ftp       419120   May 30 23:59   Duet2CombinedFirmware.bin
    -rw-rw-rw- 1 ftp ftp       402760   Dec 28 23:59   DuetMaestroFirmware.bin
    -rw-rw-rw- 1 ftp ftp       560898   Jan  1  1980   DuetWebControl.zip
    -rw-rw-rw- 1 ftp ftp       296832   Dec 28 23:59   DuetWiFiServer.bin
    -rw-rw-rw- 1 ftp ftp        63394   May 31  2019   dueui_config_default.json
    -rw-rw-rw- 1 ftp ftp        59099   May 31  2019   dueui_config.json
    -rw-rw-rw- 1 ftp ftp         1228   Jan 29 23:59   dwc2settings.json
    -rw-rw-rw- 1 ftp ftp           97   Jul 13  2018   filaments.csv
    -rw-rw-rw- 1 ftp ftp         3119   May 31  2019   heightmap.csv
    -rw-rw-rw- 1 ftp ftp          159   Jun 15  2018   homeall.g
    -rw-rw-rw- 1 ftp ftp          743   Aug 14  2017   homedelta.g
    -rw-rw-rw- 1 ftp ftp           26   Jun 13  2018   homeu.g
    -rw-rw-rw- 1 ftp ftp           26   Jun 13  2018   homev.g
    -rw-rw-rw- 1 ftp ftp          495   Jun 25  2018   homex.g
    -rw-rw-rw- 1 ftp ftp          470   Jun 25  2018   homexy.g
    -rw-rw-rw- 1 ftp ftp          495   Jun 25  2018   homey.g
    -rw-rw-rw- 1 ftp ftp          392   May 15 23:59   homez.g
    -rw-rw-rw- 1 ftp ftp        60476   Dec 28 23:59   iap4e.bin
    -rw-rw-rw- 1 ftp ftp        60748   Dec 28 23:59   iap4s.bin
    -rw-rw-rw- 1 ftp ftp         1817   Sep 14  2016   LICENSE.txt
    -rw-rw-rw- 1 ftp ftp          374   May 10 23:59   pause.g
    -rw-rw-rw- 1 ftp ftp          371   Jul 20  2018   resume.g
    -rw-rw-rw- 1 ftp ftp          147   Apr 14  2018   sleep.g
    -rw-rw-rw- 1 ftp ftp          179   Apr 14  2018   stop.g
    -rw-rw-rw- 1 ftp ftp            5   Jun 24  2018   trigger2.g
    

    Oops, meant to add... I'm running 3.0.0beta3



  • Seems like zip files are a problem. Pulling down the "www" directory has the same issue with DuetWebControl.zip but not any other files in "www".

    Geez... I forgot to say "Thanks!" for this. 🙂
    Care to create an uploader as well?



  • @gtj0 Actually you discovered a bug in RRF rather than in my tool. 😁

    When RRF sends a ZIP file it will add

    Content-Encoding: gzip
    

    to the response header which simply is wrong (except it is a renamed gzip compressed file 😉 ). I will though see if I can work around that.

    It won't be hard to create an uploader. Will sit down early next week and throw something together. 🙂


  • administrators

    There is no reason that I can see for having any .zip files on the SD card. I guess they were either copied to the card manually on a PC, or uploaded some time ago before DWC was clever enough to unzip them and upload the contained files.



  • @dc42 Still if the file is there Content-Encoding: gzip is wrong.



  • Release 1.0.1

    I just released v1.0.1 that works around the issue described by @gtj0, i.e. zip files on the SD card provoking an error. 🙂



  • is the Darwin binary supposed to work on the latest MAC os when I download it it comes as a .dms file which my mac seems to object to? (it seems to think it is a document)


  • administrators

    @wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

    @dc42 Still if the file is there Content-Encoding: gzip is wrong.

    I will look into fixing this in the 2.03 release.



  • @dougal1957 said in duetbackup - CLI tool to backup your Duet SD Card:

    is the Darwin binary supposed to work on the latest MAC os when I download it it comes as a .dms file which my mac seems to object to? (it seems to think it is a document)

    Same here.



  • @phaedrux Thank God I'm not the only one I am not that good with the intricacies as yet of the Mac and not had much cause to tbh



  • @dougal1957 @Phaedrux yes, the Darwin binary is supposed to work on MacOS 10.10 and later.

    I don't know what you tried but for clarification: this is not an installer of any kind. It's the final binary and it's a pure command line tool with no GUI.

    So after download open a shell and navigate to the directory the file sits in. Use

    chmod +x duetbackup-darwin_amd64
    

    to make it executable and then issue

    ./duetbackup-darwin_amd64
    

    You should now get an error about missing parameters - but that also means it works.

    P.S.: I'm on my phone and the filename was from memory so it might be a tiny bit different, please check that. 😉



  • @wilriker that makes more sense. I'll try again tomorrow.



  • @phaedrux let me know if it works this way.

    Also I will adjust packaging in the coming days so that a) the filename is the same fur every platform and b) it is already executable for Linux and MacOS. Right now on Windows you would also have to add .exe to the filename to achieve that - I should not do software releases when in a hurry. 🤦



  • Under windows 10 I run:

    .\duetbackup.exe -domain "192.168.0.53" -outDir ".\bkup"

    (Tried it with & without the quotes and using ./bkup and .\bkup all with same error.)

    and it errors with the following:
    2019/06/01 20:15:32 Fetching filelist for 0:/sys
    panic: time: missing Location in call to Time.In

    The whole error message is here 0_1559416600853_Duetbackup_errors.txt



  • @mudcruzr Thanks for reporting. I'll have a look into that on Monday. This part of the code is rather sketchy and even marked with a // FIXME comment already. So I guess it's time to follow my own advices. 😂



  • Release 1.0.2

    I just released version v1.0.2.

    • This release should fix the time zone issue @mudcruzr described.
    • Also starting from this release the (callable) filename for all target platforms now is duetbackup - for this the files are package as either tar.gz or zip depending on target


  • OK still not sure about this on the mac

    I tried the following with the latest 1.02 executable and get the following

    0_1559566175798_86beb62e-3019-446f-83b8-ed0598692d8f-image.png

    I suspect I need to pass some arguments to it but not sure how? any help appreciated

    Doug



  • @dougal1957 Yes, there are mandatory parameters. The names are given in the error message.

    To find out all available parameters type

    ./duetbackup --help
    

    The minimum valid command would look like

    ./duetbackup -domain "<your_duets_network_address>" -outDir <local_directory_to_save_files>
    

    This will then sync all files in the SD card's /sys (default if -dirToBackup is not provided) directory to the specified directory on your hard drive.

    P.S.: For applications implemented in Go it does not matter if you prefix parameters with one or two dashes, i.e. -domain and --domain are treated identically.



  • @wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

    Release 1.0.2

    I just released version v1.0.2.

    • This release should fix the time zone issue @mudcruzr described.
    • Also starting from this release the (callable) filename for all target platforms now is duetbackup - for this the files are package as either tar.gz or zip depending on target

    Works for me, many thanks! (Windows 10)

    I have a suggestion (only semi-seriously), if it's not too cheeky: an exclude switch, e.g.

    ./duetbackup -dirToBackup "0:/" -excludeDir "0:/gcodes" -domain "192.168.0.53" -outDir "./bkup"

    (I didn't realise how many files I had accumulated in my gcodes directory, lol)


 

Looks like your connection to Duet3D was lost, please wait while we try to reconnect.