duetbackup - CLI tool to backup your Duet SD Card [deprecated]

[wilriker]

Release 1.1.1

I had to release v.1.1.1 to fix a bug where the local directory has to exist unless the corresponding remote directory has a sub-directory.

Now all missing local directories are being created independently of the contents of the remote directory.

Also the order of file handling has been adjusted to first handle all files in the remote directory before traversing into sub-directories. This way the order of log statements makes much more sense.

[mudcruzr]

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

Release 1.1.0

I just released v1.1.0 and I think this will make @mudcruzr quite happy - as well as I hope that other users will also be happy with the new feature. Release notes below.

LOL

Happy? Happy? I'm absolutely ecstatic mate! Seriously, many thanks for a very useful tool.

[Dougal1957]

Finally got it working (Due to me not knowing what I am doing with Mac command line I might add) but does it normally take for ever to complete i normally achieve 600kib/s or more on uploads?

[Phaedrux]

@dougal1957 said in duetbackup - CLI tool to backup your Duet SD Card:

Finally got it working (Due to me not knowing what I am doing with Mac command line I might add) but does it normally take for ever to complete i normally achieve 600kib/s or more on uploads?

I went and cleaned out my Gcode directory first so that it didn't have to download all of them again.

[mudcruzr]

@mudcruzr said in duetbackup - CLI tool to backup your Duet SD Card:

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

Release 1.1.0

I just released v1.1.0 and I think this will make @mudcruzr quite happy - as well as I hope that other users will also be happy with the new feature. Release notes below.

LOL

Happy? Happy? I'm absolutely ecstatic mate! Seriously, many thanks for a very useful tool.

Ah! not so happy now even with the exclude flag it's still including all the gcode files.

Command line:

./duetbackup -domain "192.168.0.52" -outDir "Feynman" -dirToBackup "0:/" -exclude "0:/gcodes"

(Sorry!)

[gtj0]

@mudcruzr said in duetbackup - CLI tool to backup your Duet SD Card:

@mudcruzr said in duetbackup - CLI tool to backup your Duet SD Card:

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

Release 1.1.0

I just released v1.1.0 and I think this will make @mudcruzr quite happy - as well as I hope that other users will also be happy with the new feature. Release notes below.

LOL

Happy? Happy? I'm absolutely ecstatic mate! Seriously, many thanks for a very useful tool.

Ah! not so happy now even with the exclude flag it's still including all the gcode files.

Command line:

./duetbackup -domain "192.168.0.52" -outDir "Feynman" -dirToBackup "0:/" -exclude "0:/gcodes"

(Sorry!)

confirmed....

$ ./duetbackup -domain duet0 -outDir "backup" -dirToBackup "0:/" -exclude "0:/gcodes"
2019/06/04 10:42:56 Fetching filelist for 0:/
2019/06/04 10:42:57 Downloading new/changed files from 0:/ to /usr/src/3d/duet0/backup
2019/06/04 10:42:57 Fetching filelist for 0://Probe
2019/06/04 10:42:57 Downloading new/changed files from 0://Probe to /usr/src/3d/duet0/backup/Probe
2019/06/04 10:42:57 Fetching filelist for 0://System Volume Information
2019/06/04 10:42:57 Downloading new/changed files from 0://System Volume Information to /usr/src/3d/duet0/backup/System Volume Information
2019/06/04 10:42:57 Fetching filelist for 0://filaments
2019/06/04 10:42:57 Downloading new/changed files from 0://filaments to /usr/src/3d/duet0/backup/filaments
2019/06/04 10:42:57 Fetching filelist for 0://filaments/PLA
2019/06/04 10:42:57 Downloading new/changed files from 0://filaments/PLA to /usr/src/3d/duet0/backup/filaments/PLA
2019/06/04 10:42:57 Fetching filelist for 0://gcodes
2019/06/04 10:42:57 Downloading new/changed files from 0://gcodes to /usr/src/3d/duet0/backup/gcodes

Note the double //. If you specify -exclude "0://gcodes" it works.

[wilriker]

@gtj0 @mudcruzr I can see where the bug is. Will fix it tomorrow.

P.S.: Another workaround in addition to what @gtj0 already said would be to specify -dirToBackup 0: without the trailing slash.

[wilriker]

Release 1.1.2

Here we go with v1.1.2.

This release fixes the issue where trailing (and/or possibly multiple successive) slashes can break the exclude feature.

Also in this release:

• Excluding is now more efficient by checking if a directory can be excluded as a whole instead of only excluding everything inside it
• Approximate download speeds have been added to the output
• Remove local directories recursively if they no longer exist remote and -removeLocal has been given

[mloidl]

@wilriker
Hi,
Many thanks for the update, i've not tried it yet, but as i saw in the code you're now using os.RemoveAll instead of os.Remove.
This would mean that non-empty directories will be deleted too. Including e.g. .git or similar folders.

I already thought about an additional option to black list some folders/files from cleanup, but had no idea where to put that.
In my local branch (not pushed) i've created a hard-coded list but this might not be very useful for others.
Another option would be to store the filenames in a kind of config file or (.blacklist, whatever) which can be changed by users without modifying the code.

Do you have some suggestions? If you wan't i can then try to implement it.

[wilriker]

@mloidl You are right I did not think of VCS. I'll think about a better solution.

[wilriker]

Release 1.2.0

And another one named v1.2.0 has been released to address the issue brought up by @mloidl.

To prevent removing directories not managed by duetbackup the tool will now create files named .duetbackup in each directory it creates.

If for example the user will then initialize a git repository inside the local backup directory the .git directory will be skipped when removing files that do not exist remote.

[wilriker]

7 Releases in 5 days - you guys are awesome in spotting issues! I really appreciate the feedback from all of you.

[mloidl]

That was really fast.
Will test it when i'm at home on the machine.

[wilriker]

@mloidl Thanks. I borrowed the idea from tools like Dropbox or Syncthing that also use marker files (for different purposes but that doesn't matter) so it was only implementation and local testing effort.

Fun fact: while implementing everything worked immediately protecting unmanaged directories but strangely there was not a single marker file inside the directories after the tool had finished. It took me quite a while to figure out that I forgot to also exclude the marker file itself from being removed. It would even have worked this way since I create the file everytime I visit a managed directory but that would have been too much voodoo-magic-implementation to keep.

[dc42]

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

@dc42 Still if the file is there Content-Encoding: gzip is wrong.

This should be fixed in 2.03RC5 which I have just released.

[Phaedrux]

Just tested 1.2.0 on Mac using a command file to automate the process and it works great.

For those interested in making a shortcut file to run instead of using the command line each time. It will create a time stamped directory to download the backup to. You can create separate shortcut files to run different backups, different printers, etc.

This is Mac specific, but similar could be achieved on other OS as well.

Save the following into a text file and save it with .command extension. You will need to modify the following script to suit your setup

#!/bin/bash
_timestamp=$(date +"%Y-%m-%d-%H%M")
_filename="Backup-$_timestamp"
cd "/PATH/TO/DUETBACKUP/duetbackup-darwin_amd64"
./duetbackup -domain duet.local -outDir "$_filename" -dirToBackup "0:/" -exclude "0:/gcodes"

You may need to make the resulting .command file executable for it to work from the Finder. In Terminal, navigate to the directory the .command file is in, and use chmod +x FILENAME.command to make it executable so you can double click on it from the Finder to run it. Change FILENAME to be whatever the filename you saved it as.

Thanks @wilriker for the awesome backup tool!

[mloidl]

Also tested v1.2.0 on linux. Works like a charm.

Thanks @wilriker for this awesome tool.

[Exerqtor]

Uhm, might be that I'm dumber than the average in here, but i can't understand how to use this at all lol.
I'm on a Win10 system, and when i click the .exe it just open a commandline for 0.4sec and it disapears (same happens if i run it as admin).

[wilriker]

@exerqtor It's a command line tool. Clicking on it will - as you described - open the command window, show the error about missing parameters and immediately close it again.

You need to open the command line manually (open the start menu type cmd and hit enter). Then navigate to the directory the duetbackup.exe sits in and use the parameters as described in the first post or run

duetbackup --help

to get the list of available parameter with a short explanation on what their meaning is.

[Exerqtor]

Ah, thx for the quick answer.
Not gonna spend time learing to use CLI in 2019, sure a nice tool for those who allready know how to use it though.