RFM question

jens55 · 5 Oct 2023, 19:50

I would like to use the RFM tool to back up my entire SD card.
In ubuntu I do:

./rfm backup -domain 192.168.1.xxx <local backup destination path> 0:/

This does give me all the subdirectories but there are a whole bunch of other files and eventually rfm appears to hang as it is trying to fetch the file list for 0:/overlays.
I am assuming that the 0:/ directive copies not just the sd card content but somehow the firmware for the duet?
I also get a whole raft of files starting with 'bcm'. a bunch of .dat files, some .elf files, a .bin file and some .txt files

How can I just get the files on the microsd card without running rfm for each subdirectory on the SD card?

nikscha · 6 Oct 2023, 15:11

@jens55 Are you in standalone mode or SBC mode?
In SBC mode, the SD card goes into the Pi, not the duet board. There's a virtual sd card in /opt/dsf/
Regardless of mode, there's a 'Firmware' directory on the (virtual) sd card which is probably containing the files you're complaining about.
I suggest that you backup the firmware directory as well though.
I don't have an 'overlay' directory on my sd card, what's in there?

jens55 · 6 Oct 2023, 16:24

@nikscha, I use standalone mode. The SD card has no 'overlay' directory on it nor does it have the files I am seeing. Nevertheless, when using rfm and specifying the source as '0:', these files are transferred to the destination location. I haven't got a clue what they are about and was only taking a wild guess.

dc42 · 7 Oct 2023, 20:15

@jens55 in standalone mode "0:/" is the root directory on the SD card in the Duet. If you also have an external SD card socket then "1:/" is the root of that folder.

Perhaps @wilriker knows why it appears to be trying to backup a folder which doesn't exist on your SD card. Are you sure that is isn't present as a hidden directory?

jens55 · 7 Oct 2023, 21:20

@dc42, that sure is a weird one. I only have the normal SD card. I will do some more digging and also verify that there is nothing hidden on the card. Maybe there is a connection between this issue and me getting messages of getting low on memory.
Thank you for verifying that the command I am sending is correct. The microSD has more than likely been used for other stuff before. I generally wipe a card before using it but maybe I forgot to do it in this case. Taking the card out of the Duet and looking at the content did not reveal anything but I did not verify that I had the 'show hidden files' checkbox ticked in my file browser. It never occurred to me to specifically look for that.
Thanks!

jens55 · 7 Oct 2023, 23:02

@dc42, as it turned out, I did have a bunch of old crap on one of my printers microSD card which explains the odd files. BUT .... I still have the issue with rfm crashing.
I have three printers. I have tried rfm on all three printers and it consistently fails. One hangs on 0:/www/js with a "read: connection reset by peer error message and then rfm exits. On another printer, rfm hangs while "fetching filelist for 0:/sys" ... no error message, it just hangs. Printer #3 hangs while "fetching filelist for 0:www/js". This operation also just hangs.

A repeat run on all three printers had two crashes on 0:/www/js and one crash on 0:/sys.I waited maybe 5 minutes before aborting the individual instances of rfm.

One printer is connected via ethernet (6HC board) and two printers are connected via wifi (duet2wifi boards). The connection reset by peer error message is on one of the wifi boards.

I assume this is an issue with rfm and not something related to any of the duets but I thought it would be important to point this out in case people depend on rfm only to discover that the backup of the files was not complete.

jens55 · 8 Oct 2023, 21:29

Been playing around with rfm. I now believe that rfm crashes because of something encountered during the backup procedure. One printer crashes on 0:/sys, two are crashing on 0:/www/js. Each printer is consistent with it's own crash point which leads me to believe that the issue is caused by something being read during backup/file transfer.
There is one other error encountered which is 'connection reset by peer' which presumably is a network or wifi issue.

jens55 · 9 Oct 2023, 00:20

I have run into a bit of a roadblock / dead end in trying to figure out what is happening. I thought I had it figured out when reducing the subdirectories that caused rfm to crash to less than 32 Mb allowed rfm to complete it's run. This was confirmed in a number of cases but then my final test did not go as expected. I copied a 12 Mb Gcode file to the printer a total of four times, renaming it in between. According to my testing, rfm should have crashed trying to back up 0:/ when it reached the /gcodes subdirectory but it completed without an error.
At this point I don't know if maybe the 0:/gcodes subdirectory is treated differently (why would it?) or if my idea of subdirectory size causing the issue is simply wrong.
It will be up to the developer ( @wilriker ?) to carry on with the trouble shooting.

wilriker · 9 Oct 2023, 08:00

Hi @jens55

Could you run rfm with parameter -verbose. This will print out more details that might better indicate where the problem actually happens. Please post the results here then.

EDIT: I just released rfm v1.1.2-RC1 that fixes an issue with the -debug flag that will print even more details than -verbose (but otherwise has no change in behavior).

jens55 · 9 Oct 2023, 09:49

@wilriker, thanks for responding.
Using the -verbose parameter did not result in anything useful (ie there was no output, it just hung)
I installed the new version and redirected the stderr and terminal output to file output.txt
output.txt
Note that rfm hangs as it is trying to copy 0:/www/js. All the other files were already copied so this run did not copy anything (files were already present and rfm hung before it could copy anything)
Let me know if you need anything else.
Oh, I aborted rfm after maybe 30 seconds. It appeared to repeat things over and over ie it hung. I can delete the files that were backed up so that rfm actually copies something if the resulting output.txt file would help.

wilriker · 9 Oct 2023, 10:03

@jens55 Yes, that actually helped. I found the issue and will provide a fixed version soon.

The problem is that www/js on your SD card contains more files than fit into a single response and while trying to fetch the remaining entries for this directory rfm locks itself in an endless loop because there is a bug on fetching the next entries.

wilriker · 9 Oct 2023, 11:22

@jens55 Since I changed a lot of things besides fixing the issue for large file lists I bumped the version again and released the pre-release rfm v.1.2.0-RC1. Please check this. It should now work as intended. Besides the whole process can now be canceled by pressing CTRL+C cleanly.

jens55 · 9 Oct 2023, 17:04

@wilriker, Excellent! It works! Thanks for the quick fix!

wilriker · 9 Oct 2023, 18:37

@jens55 Thanks for confirming!

I edited the pre release into an actual final release of rfm 1.2.0. No need for you to switch out anything it's the exact same binaries.

Will update the tool thread tomorrow. Happy this was solved so easy and sorry for the bug.