DuetLapse3

GoremanX

I think this part needs clarification because it's contradictory the way it's currently written:

New option -createvideo (default True if omitted). The default maintains the current behavior. If False then no video is created on "restart" or "terminate" BUT will be created on "restart"

Beyond that, I like the proposed changes. It's more elegant than what I came up with. However, reading this just made me realize something:
@stuartofmt said in DuetLapse3:

Note that this needs a little more thought since I had a "todo" to address the possible situation (unhandled at the moment) of more than one instance of DuetLapse3 running against the same duet "ip".

All my duet 3D printers have the same IP address They all sit behind their own respective pi 4 computers, they all run in standalone mode with the exact same ip address, and they all write their external files to the same shared directory over NFS. So that means they would all save their photos in the same 192-168-220-2 directory I do this so that I can use the exact same Raspbian SD card image on every pi without modification. The duet boards themselves don't talk to the network, the pi computers do all the talking for them using a DHCP-assigned address.

However each printer does declare a unique hostname in config.g, even if the network isn't aware of it. Is there a way for the script to try to get the board's hostname if it hasn't been declared in the -host parameter for file/directory name purposes?

GoremanX

For fun, my first timelapse using this script:
https://youtu.be/XjBwTqbEZD8

Phaedrux

@GoremanX That was a pretty weird timelapse.

GoremanX

@Phaedrux said in DuetLapse3:

@GoremanX That was a pretty weird timelapse.

I got bored while waiting for another print to finish, and that acrylic door looked like a great screen to project silly things onto

stuartofmt

@GoremanX said in DuetLapse3:
I think this part needs clarification because it's contradictory the way it's currently written:

@stuartofmt said in DuetLapse3:

New option -createvideo (default True if omitted). The default maintains the current behavior. If False then no video is created on "restart" or "terminate" BUT will be created on "restart"

Yes - it was a brain fade .....-should read If False then no video is created on "restart" or "terminate" BUT will be created on "snapshot"

However each printer does declare a unique hostname in config.g, even if the network isn't aware of it. Is there a way for the script to try to get the board's hostname if it hasn't been declared in the -host parameter for file/directory name purposes?

Wow ! nice edge case I have some ideas, one or other of which may be "cunning". Maybe a combination of using the host computer's ip address, together with the duet address and maybe port number of DuetLapse3 (if used). Thinking cap is on.

Edit: I have a cunning plan! I will document it here tomorrow and get started on the code changes

stuartofmt

@GoremanX said in DuetLapse3:

For fun, my first timelapse using this script:
https://youtu.be/XjBwTqbEZD8

Nice! Please don't ask for embedded background images or graphics in DuetLapse3.

stuartofmt

Edit: Deleted due to changed description.

GoremanX

@stuartofmt I love it, it covers all potential use cases I can think of. And most importantly, it covers my use case, and that's all that really matters in this world. Me

stuartofmt

Version 3.3.0 has been released.

###Version 3.3.0###
[1] Made significant changes to the browser UI for both DuetLapse3 and startDuetLapse3. Both now provide buttons for easier navigation.
[2] Added an option for startDuetLapse3 to accept default start (-args) for starting DuetLapse3.

If you provide -args, special care needs to be made in formatting it correctly. See the documentation in github. Especially note
[1] double quote characters around the entire -args options list
[2] the use of

&quot;  and &apos;

for all other double or single quotes inside the outer quotes used for the -args options list.

stuartofmt

@GoremanX

Having thought about this more - and because I noticed something on WSL (which gets a different ip address each time it’s started) - the directory scheme needs a little refinement.

The problem arises when the computer ip is not static. You would end up with difficult to determine top level directories. Therefore I think using the fully qualified domain name of the computer is a better approach as it will be static. This may mean that you need - for each of your pi’s to set a different and unique domain name for each one after cloning the sd card.

Thoughts?

GoremanX

@stuartofmt Oh my pis already have different hostnames, it's dynamically assigned, so that would be perfect and even preferable. I don't do well trying to memorize the ip addresses of the hundreds of systems I interact with on a daily basis

stuartofmt

Edit: Deleted
Please see the 3.4.0 release notes below

GoremanX

@stuartofmt One observation: isn't it possible to pull the file name of the current print job through the object model? If so, instead of just processid, could the directory be named "jobfilename-processid"? That way, the directory is easily identifiable as the result of a specific job, but there's still a unique number appended to it for when the same job is run more than once

stuartofmt

@GoremanX said in DuetLapse3:

@stuartofmt One observation: isn't it possible to pull the file name of the current print job through the object model? If so, instead of just processid, could the directory be named "jobfilename-processid"? That way, the directory is easily identifiable as the result of a specific job, but there's still a unique number appended to it for when the same job is run more than once

I do not know what you mean by the jobfilename? What defines a job is the options and that would be incredibly long as a directory name. I'm pretty sure you are not proposing directory paths that look like this: ????.

/home/stuart/Lapse/
                  Stuart.localdomain/
                                    192-168-86-235/
                                                 -duet 192.168.86.235 -instances many -dontwait -port 8083 -standby -seconds 15 -detect none -camera1 stream -weburl1 http://192.168.86.230:8081/stream.mjpg

Note that the http listener provides info like this:

GoremanX

@stuartofmt Isn't this accessible through the API, same way the current machine state is?

stuartofmt

@GoremanX said in DuetLapse3:

@stuartofmt Isn't this accessible through the API, same way the current machine state is?

Ah - I see now what you mean by jobfilename. It's easy enough to get and I see how it would aid in reading the directories. There is unfortunately an issue of timing.

That the information is only available after the print job has started - so it's not guaranteed to be available when DuetLapse3 starts. Also - even when the print job does start - there are some cases (e.g. -dontwait - seconds nnn) when you could not reasonably create the directory after the job starts.

Having said that - I'll add it to the directory name (processid-jobname) if available at the time the directory is created and delay the creation of the directory until the last possible moment. i.e. immediately before the first image is captured).

Sound like a plan ?

stuartofmt

I am pleased to release Version 3.4.0. This was more difficult than I first thought. The restructuring of the directories (in particular the cleanup) introduced the need for some tight internal logic that was tricky to test.
A couple of the additional (and seemingly simple) changes ended up being quite a challenge to figure out.
The core functions were not changes so this release should be solid in that regard. Thanks to those that provided feedback and ideas on the interim design.

Enjoy.

###Version 3.4.0###
[1] Completely revised the directory and file-naming structure to facilitate many-to-many relationships between computers running DuetLapse3, Duet Printers and multiple instances of DuetLapse3.
[2] Changed how http terminate requests were handled for better cross platform compatibility.
[3] Added the ability to navigate the directory structure from a browser (new button). Also in startDuetLapse3.
[4] Added three new options: -keeplogs, -deletepics and -novideo
[5] Made some cosmetic changes to the html pages.

The new directory structure is (with repeating units [] as appropriate to your use-case)

basedir/
   |
   computername/
   |   |
   |   duetip/
   |      |
   |      |processid_jobname/         #_jobname if available
   |      |
   |      |[/processid_jobname}
   |
   |
   [computername 
     #repeat of structure above
   ]

For many (most ?) users it will simply look like this

basedir/
   |
   computername/
      |
      duetip/
         |
         |processid_jobname/         #_jobname if available

The interpretation is:
Starting from the basedir:

(1) For each computer that DuetLapse3 is running on there will be a separate directory(computername). Technically the computername will be the fully qualified domain name (FQDN). In any case - each computer needs to (and in any case should) have a unique FQDN

(2) Underneath the computername directory there will be a separate directory for each Duet that computer is connected to (duetip). All videos for this computer and duet combination go into this directory as well as the respective logfiles.

(3) Underneath the duetip directory will be temporary directories (processid_jobname).If the printjob has not started at that time - there will be no _jobname portion. This handles the the situation where multiple instances of DuetLapse3 are running on the same computer against the same Duet. This directory is created when the first image is captured.

Note that the filenames have been rationalized for this new structure. Videos and logfiles are prefixed by the processid.

To provide cross platform compatibility, colons are replaced by raised colons (more-or-less look the same). Spaces in filenames are replaced by underscore.

This is a many to many possibility that few will need but it's future proofed. This way there should be no collisions even when stored on a shared drive.

The following logic will be used to clean up files more-or-less automatically.
On startup:
[1] DuetLapse will determine how many instances are running on that computer and note their processid's. It will then delete all of the processid directories it finds (in (basedir/computername/duetip) that do not match. These will be from older (not running) instances. i.e. it will delete the old image files.
[2] If -keeplogs is NOT used. The FIRST instance of DuetLapse3 (with no other instances running) will delete the old logfiles. If -keeplogs is used then the logfiles will not be deleted. This means that if you typically use -keeplogs you can affect a cleanup by running a single instance of DuetLapse3 without -keeplogs.

On Terminate
if -deletepics is used. The basedir/computername/duetip/processid directory will be deleted (i.e. the images will be deleted).

The new option -novideo will prevent a video being created when DuetLapse3 terminates. If BOTH -novideo and -deletepics are specified- this means that if you want to use the images you need to have done so before terminating that instance of DuetLapse3. For example using the snapshot feature provided by the html listener.

GoremanX

@stuartofmt Very cool, great job! You just covered all of the complexities of my crazy use-case, and that's no small feat I look forward to trying it out as soon as my warranty replacement Duet 3 board arrives... which should be any month now...

JohnOCFII

I'm looking forward to trying this new version. I haven't moved beyond the original yet, and I've got dying process issues on my Pi that might be related to how I hacked that together.

John

stuartofmt

@JohnOCFII said in DuetLapse3:

I'm looking forward to trying this new version. I haven't moved beyond the original yet, and I've got dying process issues on my Pi that might be related to how I hacked that together.

John

I have put in quite a lot of defensive code compared to the original. So this version should, in general, be more robust. As a quick datum - last night I ran three instances concurrently on a Pi 3B+ each with a browser tab polling status every 60 seconds and as well startDuetLapse3 running. Each instance captured images every 60 seconds from the camera (also running on the Pi). So a reasonable "soak test". Each captured over 750 images (~ 2200 in total).

I then issued a "terminate all" from startDuetLapse3 and only two videos were successfully created. The problem was an out-of-memory error from ffmpeg. I've not seen this type of error before and guess it was three instances of ffmeg trying all at once to create a video. I'm going to try and increase the swapfile to see if that changes anything. The images are all there - so it is recoverable.

As an aside I have startDuetLapse3 running under systemctl with restart - so it is continuously running and will restart after a shutdown - so its "always there"