DuetLapse3

GoremanX

@stuartofmt I love it, it covers all potential use cases I can think of. And most importantly, it covers my use case, and that's all that really matters in this world. Me

stuartofmt

Version 3.3.0 has been released.

###Version 3.3.0###
[1] Made significant changes to the browser UI for both DuetLapse3 and startDuetLapse3. Both now provide buttons for easier navigation.
[2] Added an option for startDuetLapse3 to accept default start (-args) for starting DuetLapse3.

If you provide -args, special care needs to be made in formatting it correctly. See the documentation in github. Especially note
[1] double quote characters around the entire -args options list
[2] the use of

&quot;  and &apos;

for all other double or single quotes inside the outer quotes used for the -args options list.

stuartofmt

@GoremanX

Having thought about this more - and because I noticed something on WSL (which gets a different ip address each time it’s started) - the directory scheme needs a little refinement.

The problem arises when the computer ip is not static. You would end up with difficult to determine top level directories. Therefore I think using the fully qualified domain name of the computer is a better approach as it will be static. This may mean that you need - for each of your pi’s to set a different and unique domain name for each one after cloning the sd card.

Thoughts?

GoremanX

@stuartofmt Oh my pis already have different hostnames, it's dynamically assigned, so that would be perfect and even preferable. I don't do well trying to memorize the ip addresses of the hundreds of systems I interact with on a daily basis

stuartofmt

Edit: Deleted
Please see the 3.4.0 release notes below

GoremanX

@stuartofmt One observation: isn't it possible to pull the file name of the current print job through the object model? If so, instead of just processid, could the directory be named "jobfilename-processid"? That way, the directory is easily identifiable as the result of a specific job, but there's still a unique number appended to it for when the same job is run more than once

stuartofmt

@GoremanX said in DuetLapse3:

@stuartofmt One observation: isn't it possible to pull the file name of the current print job through the object model? If so, instead of just processid, could the directory be named "jobfilename-processid"? That way, the directory is easily identifiable as the result of a specific job, but there's still a unique number appended to it for when the same job is run more than once

I do not know what you mean by the jobfilename? What defines a job is the options and that would be incredibly long as a directory name. I'm pretty sure you are not proposing directory paths that look like this: ????.

/home/stuart/Lapse/
                  Stuart.localdomain/
                                    192-168-86-235/
                                                 -duet 192.168.86.235 -instances many -dontwait -port 8083 -standby -seconds 15 -detect none -camera1 stream -weburl1 http://192.168.86.230:8081/stream.mjpg

Note that the http listener provides info like this:

GoremanX

@stuartofmt Isn't this accessible through the API, same way the current machine state is?

stuartofmt

@GoremanX said in DuetLapse3:

@stuartofmt Isn't this accessible through the API, same way the current machine state is?

Ah - I see now what you mean by jobfilename. It's easy enough to get and I see how it would aid in reading the directories. There is unfortunately an issue of timing.

That the information is only available after the print job has started - so it's not guaranteed to be available when DuetLapse3 starts. Also - even when the print job does start - there are some cases (e.g. -dontwait - seconds nnn) when you could not reasonably create the directory after the job starts.

Having said that - I'll add it to the directory name (processid-jobname) if available at the time the directory is created and delay the creation of the directory until the last possible moment. i.e. immediately before the first image is captured).

Sound like a plan ?

stuartofmt

I am pleased to release Version 3.4.0. This was more difficult than I first thought. The restructuring of the directories (in particular the cleanup) introduced the need for some tight internal logic that was tricky to test.
A couple of the additional (and seemingly simple) changes ended up being quite a challenge to figure out.
The core functions were not changes so this release should be solid in that regard. Thanks to those that provided feedback and ideas on the interim design.

Enjoy.

###Version 3.4.0###
[1] Completely revised the directory and file-naming structure to facilitate many-to-many relationships between computers running DuetLapse3, Duet Printers and multiple instances of DuetLapse3.
[2] Changed how http terminate requests were handled for better cross platform compatibility.
[3] Added the ability to navigate the directory structure from a browser (new button). Also in startDuetLapse3.
[4] Added three new options: -keeplogs, -deletepics and -novideo
[5] Made some cosmetic changes to the html pages.

The new directory structure is (with repeating units [] as appropriate to your use-case)

basedir/
   |
   computername/
   |   |
   |   duetip/
   |      |
   |      |processid_jobname/         #_jobname if available
   |      |
   |      |[/processid_jobname}
   |
   |
   [computername 
     #repeat of structure above
   ]

For many (most ?) users it will simply look like this

basedir/
   |
   computername/
      |
      duetip/
         |
         |processid_jobname/         #_jobname if available

The interpretation is:
Starting from the basedir:

(1) For each computer that DuetLapse3 is running on there will be a separate directory(computername). Technically the computername will be the fully qualified domain name (FQDN). In any case - each computer needs to (and in any case should) have a unique FQDN

(2) Underneath the computername directory there will be a separate directory for each Duet that computer is connected to (duetip). All videos for this computer and duet combination go into this directory as well as the respective logfiles.

(3) Underneath the duetip directory will be temporary directories (processid_jobname).If the printjob has not started at that time - there will be no _jobname portion. This handles the the situation where multiple instances of DuetLapse3 are running on the same computer against the same Duet. This directory is created when the first image is captured.

Note that the filenames have been rationalized for this new structure. Videos and logfiles are prefixed by the processid.

To provide cross platform compatibility, colons are replaced by raised colons (more-or-less look the same). Spaces in filenames are replaced by underscore.

This is a many to many possibility that few will need but it's future proofed. This way there should be no collisions even when stored on a shared drive.

The following logic will be used to clean up files more-or-less automatically.
On startup:
[1] DuetLapse will determine how many instances are running on that computer and note their processid's. It will then delete all of the processid directories it finds (in (basedir/computername/duetip) that do not match. These will be from older (not running) instances. i.e. it will delete the old image files.
[2] If -keeplogs is NOT used. The FIRST instance of DuetLapse3 (with no other instances running) will delete the old logfiles. If -keeplogs is used then the logfiles will not be deleted. This means that if you typically use -keeplogs you can affect a cleanup by running a single instance of DuetLapse3 without -keeplogs.

On Terminate
if -deletepics is used. The basedir/computername/duetip/processid directory will be deleted (i.e. the images will be deleted).

The new option -novideo will prevent a video being created when DuetLapse3 terminates. If BOTH -novideo and -deletepics are specified- this means that if you want to use the images you need to have done so before terminating that instance of DuetLapse3. For example using the snapshot feature provided by the html listener.

GoremanX

@stuartofmt Very cool, great job! You just covered all of the complexities of my crazy use-case, and that's no small feat I look forward to trying it out as soon as my warranty replacement Duet 3 board arrives... which should be any month now...

JohnOCFII

I'm looking forward to trying this new version. I haven't moved beyond the original yet, and I've got dying process issues on my Pi that might be related to how I hacked that together.

John

stuartofmt

@JohnOCFII said in DuetLapse3:

I'm looking forward to trying this new version. I haven't moved beyond the original yet, and I've got dying process issues on my Pi that might be related to how I hacked that together.

John

I have put in quite a lot of defensive code compared to the original. So this version should, in general, be more robust. As a quick datum - last night I ran three instances concurrently on a Pi 3B+ each with a browser tab polling status every 60 seconds and as well startDuetLapse3 running. Each instance captured images every 60 seconds from the camera (also running on the Pi). So a reasonable "soak test". Each captured over 750 images (~ 2200 in total).

I then issued a "terminate all" from startDuetLapse3 and only two videos were successfully created. The problem was an out-of-memory error from ffmpeg. I've not seen this type of error before and guess it was three instances of ffmeg trying all at once to create a video. I'm going to try and increase the swapfile to see if that changes anything. The images are all there - so it is recoverable.

As an aside I have startDuetLapse3 running under systemctl with restart - so it is continuously running and will restart after a shutdown - so its "always there"

stuartofmt

@stuartofmt said in DuetLapse3:
I'm going to try and increase the swapfile to see if that changes anything. The images are all there - so it is recoverable.

I increased the swapfile and that does seem to have solved the issue. Success with 3 instances, each with over 1300 images (total ~ 4000). The Pi I tested on only had 100M of swapfile. I increased it to 1024M. During the test ~ 350M was used. I'm on Debian Buster so the change was an edit to /etc/dphys-swapfile followed by a reboot. I did check but there does not seem to be a simple way to limit memory usage in ffmpeg - so this looks like the way to go.

GoremanX

@stuartofmt Looks like I need to update the master image for my Pis! Out of curiosity, how much memory is on the boards you've been testing with? I'm a cheap bastard, I tend to stick to the base boards, so all mine have 1GB (older) or 2GB (current). I've been considering getting some fancy-schmancy 8GB boards for the next ones, but it's hard to justify that $40 premium just for memory I might never make use of

stuartofmt

@GoremanX said in DuetLapse3:

Out of curiosity, how much memory is on the boards you've been testing with?

From

cat /proc/cpuinfo | grep Model

Model                 : Raspberry Pi 3 Model B Rev 1.2

This model has:
1.2 GHz 64-bit quad core ARM Cortex-A53 and 1GB RAM
I have a 32GB card.
In all - it seems to cope well. Here is what it looks like with the following load:

Camera streaming - 9 simultaneous connections (3 internal - 6 external taking snapshots ~ 2 per second) + 1 external constantly streaming connection.
startDuetLap connectionse3
DuetLapse3 - 3 instances capturing images every 5 seconds
chromium - running dueui

Chromium is the biggest load ~ 17%. ffmpeg slips in periodically at about 12% and the DuetLapse3 instances each peak at 7% when they are busy .

The load average suggests this is handling it pretty well (lets say ~ 25% CPU utilization).

But as you can see - the previous 100M swapfile was not enough. 1GB may be a little big but it seems like it will be a "safe" number.

This is further supported by:

pi:~ $ uptime
 11:15:16 up 18:58,  3 users,  load average: 1.21, 1.08, 0.98

GoremanX

Finally got a chance to try this yesterday. It works perfect. I'm using reprap_notify to monitor the state of the printer. It polls the Duet 3 board every 30 seconds. When it detects that a print job has started, it starts an instance of DuetLapse3. When the job ends, it sends a terminate command via curl. I get an email notification of each event, and also if/when the printer pauses. All the files end up exactly where they need to. I have a second browser tab for monitoring the status of DuetLapse3. Everything appears to run reliably.

stuartofmt

@GoremanX said in DuetLapse3:

It works perfect.

Thanks for the feedback!
I'm sure you have not tried hard enough to break it

I do have a small update coming but this is just to protect against some ffmpeg limitations. ffmpeg tries to gobble up lots of CPU and Memory if multiple instances are running. Does not affect most users but surfaced in my stress testing.

stuartofmt

@GoremanX said in DuetLapse3:

When it detects that a print job has started, it starts an instance of DuetLapse3.

I'm curious as to the utility of this approach versus DuetLapse3 just sitting there waiting for a print job to start (-detect layer) ? I'm asking as it may suggest missing functionality or improvements.

@GoremanX said in DuetLapse3:

When the job ends, it sends a terminate command via curl.

DuetLapse3 will terminate itself when the print job completes - so wondering why you are sending a separate command ? A bug in a prior release ?

GoremanX

@stuartofmt I found that the layer detection starts capture too late sometimes. I prefer to use "dontwait" and start the process manually, even if it results in capturing initial Calibration and bed probing for nothing. I'd rather have too many photos at the beginning than not enough.

As for terminating manually, that's mostly to be 100% sure all processes got terminated. I send pid=all because in my case, a specific pi only ever monitors a single printer at a time. But having said that, I've also found that capturing ends a tad too soon. I never get a capture with the print head out of frame once the print is complete, which would be nice to have as the final frame of the video. Rather than end at the end of the last layer, it would be nice if it ended once the job is actually complete and took at least one last photo at that point before terminating