SBC plugin to backup Duet3d files - duetBackup

[stuartofmt]

This plugin provides periodic or instant backups of duet3d SBC files. Backups are made to a Github repository, thereby facilitating the ability to roll back to earlier file versions.

Modes of operation
There are two modes of operation, depending on whether the archive option is used.

Files deleted from the source are deleted from the main branch (default)
The designated directories are compared to the files in the main branch. Files are either Added (new files), Updated (if changed), or Skipped (if not changed). Files that have been deleted from the designated directories are removed from the main branch. In this way, the main branch is a snapshot of the designated directories.

If you need to recover to the latest version - a simple download of the top level main branch is all that is needed.

Github history still provides access to previously deleted files.

Files are not deleted from the main branch
The behavior is the same as above, except that files which are deleted from the source are NOT deleted from the main branch. This makes recovery a little more involved.

Prerequisites:
SBC

V3.5.+ or V3.6+

Tested with Debian Bullseye and Bookworm

Python3

To use the plugin:

1- Create a new repository (Private is recomended) on Github. Include a Readme.md file. Take note of the name of the repository (including case) as well as the branch, which will usually be main.

2- Create a personal access token token. A classic token with "repo" authorization is sufficient. Instructions for creating a token can be found here:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens

3- Create(using DWC) a file System/duetBackup/duetBackup.config details are in the file Config Notes.

4- Install the plugin using the plugin.zip file from here:
https://github.com/stuartofmt/duetBackup

5- To create an initial backup, set -days 0 and -hours 0. This will run duetBackup once. After that, set -days and -hrs to your prefered backup interval. Note that if there are no changes to any files, backup does nothing.

Versions

V1.0

• Initial release

V1.1

• Added messages, sent to DWC
• Displays time to next backup using local time (previously GMT)
• Added -duetPassword to support for printers that use a password
• Added -verbose to enable more detailed log messages.

V1.2

• Added -noDelete option
• Compares files to determine if an update is needed.
• Added date / time of last change to file (except initial save).

V1.3

• README.md file shows date and time of last backup
• README.md prevented from being deleted
• Added -ignore. Can specify files that are not to be backed up. This also causes these files to be deleted unless -noDelete is set.

[stuartofmt]

Added some embelishments and tidy up.

V1.1

• Added some status messages, sent to DWC
• Next backup time is displayed using local time (previously GMT)
• Added -duetPassword for folks that use a password
• Added -verbose to enable more detailed log messages.

[stuartofmt]

V1.2

Some improvements that facilitate two modes of operation (see desription in iniial post above). Better identification of when individual files were Added, Updated, etc

• Added -noDelete option
• Compares files to determine if an update is needed.
• Added date / time of last change to file (except initial save).

[stuartofmt]

V1.3

• README.md file shows date and time of last backup
• README.md prevented from being deleted
• Added -ignore. Can specify files that are not to be backed up. This also causes these files to be deleted unless -noDelete is set.

[chrishamm]

@stuartofmt Thanks for your effort, please consider submitting it to https://plugins.duet3d.com/ as well

[stuartofmt]

@chrishamm said in SBC plugin to backup Duet3d files:

@stuartofmt Thanks for your effort, please consider submitting it to https://plugins.duet3d.com/ as well

Hi. I have tried to submit it but it gets summarily rejected. My guess is that its looking for a specific folder structure on github ... (or something) ... The submission instructions are unclear on this aspect.

I have raised an issue (twice) on github. I asked about the problem on the forum (11 days ago) and it was refered to @yasas but there has been no reply.

[chrishamm]

@stuartofmt Oh, sorry to hear, I'll make sure this is addressed ASAP.

[chrishamm]

@stuartofmt The reason why it cannot be approved is because there is no GitHub release on your repository:

You must create one before you can submit it on the plugin repository. Note that this release must contain your plugin ZIP since that is linked by the plugin repo.

[yasas]

@stuartofmt Hi @stuartofmt I'm sorry to hear this I have missed the GH issues as for some reason they have been closed/wasn't notified. I will update the error messages in case someone else runs into the same issue.

[jltx]

@stuartofmt I am on 3.5.4 with SBC on Bookworm RPi. I cannot install the plug-in due to the following error:

Failed to install package PyGithub==2.1.1, package manager exited with code 67

I tried to apt install it but apt cannot find it in the default repos.

[stuartofmt]

@jltx

TLDR; You need to be using Duet V3.6.0+ (or not use Bookworm)

Bookworm introduced some new requirements for installing python programs and modules. They need to be installed in a virtual environment. In practice, this means a virtual environment for each plugin (so they don't stomp on each other).
The plugin manager in V3.6 uses an installer that takes care of this, version 3.5 does not.

Unless installing deliberately into the system environment - you should not be using apt for python packages. I suspect there are other safeguards in place as well.

On my system (SBC, Bookworm, Duet 3.6.0 - beta.4) - trying to install into the system environment, using pip, I get the expected warning

pi@srsender:~ $ python -m pip install "PyGithub=="
error: externally-managed-environment

This environment is externally managed
    To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

The required version of pyGithub is however available:

pi@srsender:~ $ python -m pip index versions PyGithub
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
PyGithub (2.5.0)
Available versions: 2.5.0, 2.4.0, 2.3.0, 2.2.0, 2.1.1, 2.1.0.post0, 1.59.1, 1.59.0, 1.58.2, 1.58.1, 1.58.0, 1.57, 1.56, 1.55, 1.54.1, 1.54.0.1, 1.54, 1.53, 1.52, 1.51, 1.50, 1.47, 1.46, 1.45, 1.44.1, 1.44, 1.43.8, 1.43.7, 1.43.6, 1.43.5, 1.43.4, 1.43.3, 1.43.2, 1.43.1, 1.43, 1.40, 1.39, 1.38, 1.37, 1.36, 1.35, 1.34, 1.33, 1.32, 1.31, 1.30, 1.29, 1.28, 1.27.1, 1.26.0, 1.25.2, 1.25.1, 1.25.0, 1.24.1, 1.24.0, 1.23.0, 1.22.0, 1.21.0, 1.20.0, 1.19.0, 1.18.0, 1.17.0, 1.16.0, 1.15.0, 1.14.2, 1.14.1, 1.14.0, 1.13.1, 1.13.0, 1.12.2, 1.12.1, 1.11.1, 1.11.0, 1.10.0, 1.9.1, 1.9.0, 1.8.1, 1.8.0, 1.7, 1.6, 1.5, 1.4, 1.3, 1.2, 1.1, 1.0, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0.0

[jltx]

@stuartofmt Thanks. I ran into the venv elsewhere so that makes sense. I will move to 3.6.