DuetLapse3
-
Released version 3.4.1. Much nicer to use the browser UI with some extra "goodies" to avoid the need to remote into your server. Some small functional tweaks based on feedback from the last release.
###Version 3.4.1###
Changes to DuetLapse3
[1] Changed the browser UI to a single page layout.
[2] The file function is restricted to the specific instance.
[3] File functions expanded to allow deletion of video files. startDuetLapse3 has more options.
[4] If using -detect layer capture starts on layer 0 (previously was layer 1).
[5] An additional image is captured immediately before a video is created, independent of other settings.
[6] If the version of ffmpeg does not support -extratime it is ignored.Changes to startDuetLapse3
[1] Changed the browser UI to a single page layout.
[2] Added an optional argument (-topdir) to set the top level directory for file functions. If used - this would normally be set the same as DuetLapse or at the "duetip" level
[3] File functions expanded to allow delete and zip. This is "conservative" - will not allow deletion of files / directories of running instances. Can only zip directories.Note: If it was not obvious - Clicking on the links either opens, displays or downloads the directory / file depending on your browser settings.
-
Finally getting around to replacing Danel's version.
I'm running this on CentOS Stream 8.
I received this error on my first test run. I haven't troubleshot yet, as I've got to head out the door, but looks like a character set/codec issue.
My command line:
python3 /opt/DuetLapse/DuetLapse3.py -duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect none -dontwait -seconds 2 -extratime 4My error message:
pi@octocore:~ $ python3 /opt/DuetLapse/DuetLapse3.py -duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect none -dontwait -seconds 2 -extratime 4 Cleaning up phase: startup Traceback (most recent call last): File "/opt/DuetLapse/DuetLapse3.py", line 1732, in <module> init() File "/opt/DuetLapse/DuetLapse3.py", line 225, in init f_handler = logging.FileHandler(logfilename, mode='w') File "/usr/lib/python3.7/logging/__init__.py", line 1092, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python3.7/logging/__init__.py", line 1121, in _open return open(self.baseFilename, self.mode, encoding=self.encoding) UnicodeEncodeError: 'latin-1' codec can't encode character '\u02f8' in position 61: ordinal not in range(256) pi@octocore:~ $ -
Looking a little more closely, issue seems to be in here:
-weburl1 http://octocore.localdomain:8081/snapshotIf I change weburl1 to weburl I get a different error.
Could this be a permissions issue? I installed DuetLapse3 in /opt. I see it did create subdirectories like so:
pi@octocore:/opt $ tree DuetLapse/ DuetLapse/ |-- DuetLapse3.py `-- octocore `-- railcore-localdomain 2 directories, 1 file pi@octocore:/opt $ -
@JohnOCFII said in DuetLapse3:
If I change weburl1 to weburl I get a different error.
Hi John - We'll get you going.
The first thing is that it's weburl1 (numerical 1 at the end). Putting weburl will result in an error message about an unrecognized argument.The other error is complaining because it looks like your OS is set to latin-1 whereas pretty much any code written today (inclusing DuetLapse3) anticipates utf8. The reason for that particular character (raised colon) is because Windows does not allow a regular colon in filenames and visually - colons are useful for time representation .... so I used raised colons for cross platform consistency
The area in the code where the issue is being raised looks to be quite early - setting up the logfile and I suspect the OS is complaining at that point.
Are you willing to change your Centos settings?
This seems to suggest that its pretty straightforward.
https://serverfault.com/questions/275403/how-do-i-change-my-locale-to-utf-8-in-centosEdit: It could also be the way the filesystem is mounted. Take a look at /etc/fstab - there may be some limiting settings there ....
And this looks like it may be useful:
https://www.osetc.com/en/centos-7-rhel-7-change-the-system-locale.html#:~:text=To show the current locale,βcat %2Fetc%2Flocale.For reference - this is what my Debian systems look like:
pi@srsenderpi:~ $ localectl status System Locale: LANG=en_US.UTF-8 VC Keymap: n/a X11 Layout: us X11 Model: pc105 -
Actually, I lied. I have a bunch of CentOS VMs, but THIS is running on a Raspberry Pi.
pi@octocore:/opt $ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debianBut somehow, the locale is not correct:
pi@octocore:/opt $ localectl status System Locale: LANG=en_US VC Keymap: n/a X11 Layout: gb X11 Model: pc105 pi@octocore:/opt $I'm running through raspi-config now to reset locale.
I'll report back soon! -
@stuartofmt Fixing the locale to be UTF-8 compliant got me past that issue -- thanks!
I also had installed DuetLapse in /opt, and had forgotten to set the ownership to user pi, so I was having some permission issues, but that was easy to spot and to fix.
Basic test went fine. Now on to setting up and testing
startDuetLapse3. -
@JohnOCFII said in DuetLapse3:
Basic test went fine. Now on to setting up and testing
startDuetLapse3.Great to hear !
Note that there are a couple of small differences between DuetLapse3 and the original. Mainly in the more exotic settings. The defaults Omitted options) should result in more-or-less the same behavior.
I think you will like startDuetLapse3. You can then do most everything from a browser without "gymnastics".
Appreciate any and all feedback.
-
@stuartofmt So far, so good. About 8 short prints so far today.
One interesting observation. I use multiple subdirectories for my GCODE files on my printer. Up until this last print, all the DuetLapse tests I've done were from one directory. I just did a small test from another directory (at the same level in the printer GCODE files hierarchy), yet it appears DuetLapse created a separate subdirectory for this job, and created and kept all the JPEGs for this last movie.
Here is the directory tree:
pi@octocore:/opt/DuetLapse $ tree . βββ DuetLapse3.py βββ octocore βββ railcore-localdomain βββ 4974_Camera1_Sun-10-51.mp4 βββ 5343_Camera1_Sun-12-18.mp4 βββ 5868_Camera1_Sun-14-59.mp4 βββ 6261_21-03-21T18ΛΈ12ΛΈ26.log βββ 6261_Calibration βΒ Β βββ HoseSideRingTest-v3_0.2mm_PETG_RC βΒ Β βββ Camera1_00000001.jpeg βΒ Β βββ Camera1_00000002.jpeg βΒ Β βββ Camera1_00000003.jpeg βΒ Β βββ Camera1_00000004.jpeg βΒ Β βββ Camera1_00000005.jpeg βΒ Β βββ Camera1_00000006.jpeg βΒ Β βββ Camera1_00000007.jpeg βΒ Β βββ Camera1_00000008.jpeg βΒ Β βββ Camera1_00000009.jpeg βΒ Β βββ Camera1_00000010.jpeg βΒ Β βββ Camera1_00000011.jpeg βΒ Β βββ Camera1_00000012.jpeg βΒ Β βββ Camera1_00000013.jpeg βΒ Β βββ Camera1_00000014.jpeg βΒ Β βββ Camera1_00000015.jpeg βΒ Β βββ Camera1_00000016.jpeg βΒ Β βββ Camera1_00000017.jpeg βΒ Β βββ Camera1_00000018.jpeg βΒ Β βββ Camera1_00000019.jpeg βΒ Β βββ Camera1_00000020.jpeg βΒ Β βββ Camera1_00000021.jpeg βΒ Β βββ Camera1_00000022.jpeg βββ 6261_Camera1_Sun-18-23.mp4 βββ 812_Camera1_Sat-21-57.mp4 4 directories, 29 files pi@octocore:/opt/DuetLapse $ -
@JohnOCFII said in DuetLapse3:
@stuartofmt So far, so good. About 8 short prints so far today.
One interesting observation. I use multiple subdirectories for my GCODE files on my printer. Up until this last print, all the DuetLapse tests I've done were from one directory.
So it looks like the print job was named /Calibration/HoseSideRingTest-v3_0.2mm_PETG_RC Does this seem right given your situation ? Code-wise, the print job name is appended to the process id and a directory created accordingly.
If this is not the case then I'd need more info to make sense of it - specifically
(1) The options used to start DuetLapse3
(2) The hostname of the computer running DuetLapse3 (looks to be octocore)
(2) the duet printer name (looks to be railcore-localdomain)
(3) the full printjobname and the gcode directory in which it was placed.Are the directories cleaned up when you start a new (only one running) instance of DuetLapse3?
Take a look at the documentation and see if its clear. If not - let me know and I will fix.
EDIT: I can probably (and likely will) suppress the possibility of higher level directory names - but in any case understanding your situation will be helpful.
-
@stuartofmt said in DuetLapse3:
So it looks like the print job was named /Calibration/HoseSideRingTest-v3_0.2mm_PETG_RC Does this seem right given your situation ? Code-wise, the print job name is appended to the process id and a directory created accordingly.
This is correct.
Are the directories cleaned up when you start a new (only one running) instance of DuetLapse3?
Yes -- I think what confused me is that at the time I created that tree, the directory and JPEGs were still around -- I was thinking they got cleaned up at the end of the current job, and not the start of the next job.
I actually prefer your current design.
Also - I am really glad of your current design for another reason -- I love the log file! I ran a 7.5 hour print yesterday evening, and my Pi lost network connectivity 5 times during the print. All the successfully captured images are still available for me to re-run through FFMPEG! Separately I need to try to use the wired Ethernet instead of WiFi to see if the WiFi on this particular Pi is the issue.
-
@JohnOCFII said in DuetLapse3:
I ran a 7.5 hour print yesterday evening, and my Pi lost network connectivity 5 times during the print. All the successfully captured images are still available for me to re-run through FFMPEG! Separately I need to try to use the wired Ethernet instead of WiFi to see if the WiFi on this particular Pi is the issue.
I mostly use a Pi with my Duet2 Wifi. Both are fairly close to the router - so good connectivity. Nonetheless - DuetLapse3 should (provided the connectivity issues do not last too long) work through them. I tested the code by deliberately turning off the Duet , performing multiple emergency restarts etc. In short - with a bit of luck - DuetLapse3 should create the video provided it saw the transition to "idle" at the end of the print job. Note that today I discovered a possible "transient status" that may prevent a video being created at the end of the print job. It's possibly related to my custom ending gcode - but I have a fix
In any case - "yes", that's a good reason for leaving the captured images in place at the end of the job. Also - if you have not noticed - you can use startDuetLapse3 to zip up the directory (after the DuetLapse3 instances has terminated) then download the resultant zipfile, all from a browser.
I'm thinking about adding a "create video" button to simplify the recovery - but at some point - I've go to stop adding in "stuff".
-
@stuartofmt It is great to see how your mind works on this stuff!
FYI -- I did disable WiFi, and switch to wired Ethernet. I am no longer seeing connectivity issues, but Duetlapse is still seeing "something" network related. I will gladly 100% blame this on the weak networking code in the firmware on the Duet2 -- whatever library is used in this hardware (not from dc42 -- but whatever the vendor provides) seems to act as if it hasn't seen changes to networking code since the 1990's.
This is what I see a few times an hour:
railcore.localdomain !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! railcore.localdomain There was a network failure: HTTPConnectionPool(host='railcore.localdomain', port=80): Max retries exceeded with url: /rr_model?key=state.status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x764ad170>: Failed to establish a new connection: [Errno 111] Connection refused')) railcore.localdomain !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! railcore.localdomainor
railcore.localdomain !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! railcore.localdomain There was a network failure: HTTPConnectionPool(host='railcore.localdomain', port=80): Max retries exceeded with url: /rr_model?key=state.status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7664cdf0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) railcore.localdomain !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! railcore.localdomain railcore.localdomain getDuetStatus failed to get data. Code: 9999 Reason: Connection Error railcore.localdomain Printer is disconnected - Trying to reconnect railcore.localdomain ****** Duet status changed to: processing ***** railcore.localdomain ****** Print State changed to: Capturing *****As you suggested, DuetLapse seems to work past it without issue.
John
-
The problem is that DuetLapse3 is asking for the current status and not getting a reply. It dutifully retries after a brief pause but will eventually give up. For the most part - it can work through and it does not matter if it misses capturing an image.
It may be pure network related or it mat be that DWC is "overloaded" servicing too many requests per unit time. If you have other applications getting status frequently (lets say every second) then that might be an issue - but other than that - 1990s network code seems as good a guess as any.
-
I'd guess that running DWC on two other devices in addition to DuetLapse might have had the Duet2 board confused...
Today, after a fresh start on the printer and after quitting DWC after the first layer completed, I didn't see any disconnect errors.
Here's a nice example DuetLapse3 timelapse. This was a 3 hour 20 minute print in a 53 second timelapse.
-
Still figuring stuff out for my setup, but this is getting fun
-
@GoremanX said in DuetLapse3:
Still figuring stuff out for my setup, but this is getting fun
Really raising the bar with music... I'll have to see what I can find!
-
@JohnOCFII said in DuetLapse3:
Really raising the bar with music... I'll have to see what I can find!
Hah! "Stock" previews from Adobe Premiere Pro. Not even the fully licensed track, just the so-called low quality preview version. If enough people view the video, YouTube will flag it, but it's fine to use for a limited expected audience
-
@stuartofmt I'd like some guidance before I head off in the wrong direction.
I've set up (but not yet started):
/etc/systemd/system/StartDuetLapse3.servicewhich calls:
/opt/DuetLapse/StartDuetLapse3.pywhich calls:
/opt/DuetLapse/DuetLapse3.pyNow -- when I've been testing this over the last few days manually (without the systemd service, and without using StartDuetLapse3) I've used the following command line:
python3 /opt/DuetLapse/DuetLapse3.py -duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect layer -extratime 4So switching to this service-initiated launch, do I put all the arguments in the systemd service file like this?
ExecStart=python3 /opt/DuetLapse/StartDuetLapse3.py -port 8082 -args="-duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect layer -extratime 4"Thanks for your review!
John
-
@JohnOCFII said in DuetLapse3:
So switching to this service-initiated launch, do I put all the arguments in the systemd service file like this?
ExecStart=python3 /opt/DuetLapse/StartDuetLapse3.py -port 8082 -args="-duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect layer -extratime 4"Essentially - yes but note:
(1) it looks like you have capitalized startDuetLapse3.py
(2) startDuetLapse3 does not directly call DuetLapse3. Its a browser accessed service that provides various controls. The "args=" option sets a default string (that you can edit) in the browser interface. For example my startDuetLapse3.service file looks like this:[Unit] Description=duetLapse3 Service After=multi-user.target [Service] WorkingDirectory=/home/pi/Lapse User=pi Type=idle ExecStart=python3 /home/pi/Lapse/startDuetLapse3.py -port 8082 -topdir /home/pi/Lapse/Captures -args=" -duet 192.168.86.235 -port 8083 -basedir /home/pi/Lapse/Captures -dontwait -seconds 15 -detect none -weburl1 http://192.168.86.230:8081/stream.mjpg -camera1 other -camparam1="'ffmpeg' +ffmpegquiet + ' -y -i ' +weburl+ ' -vframes 1 ' +fn+debug"" Restart=always [Install] WantedBy=multi-user.targetand results in the browser showing this (when start is pressed).
The text above the second start is what is extracted from args=. It's editable (or you could leave out args= and type / copy / paste into that field.
The important thing is its not until the second "start" is pressed that DUetLapse3 is started. In this manner you can easily manage multiple instances if you wish.
Hope this helps.
P.S. I've made a few tweaks. likely out in a day or two. Mainly the ability to create video from startDuetLapse3 (if the dir still exists).
-
@stuartofmt Thanks for the review.
Checking status now, I get the following message:
pi@octocore:/etc/systemd/system $ sudo systemctl status startDuetLapse3.service β startDuetLapse3.service - DuetLapse3 timelapse launch script Loaded: loaded (/etc/systemd/system/startDuetLapse3.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-03-25 19:51:17 CDT; 13h ago Main PID: 471 (python3) Tasks: 2 (limit: 2062) CGroup: /system.slice/startDuetLapse3.service ββ471 /usr/bin/python3 /opt/DuetLapse/startDuetLapse3.py -port 8082 Mar 26 04:10:03 octocore python3[471]: method() Mar 26 04:10:03 octocore python3[471]: File "/opt/DuetLapse/startDuetLapse3.py", line 198, in do_GET Mar 26 04:10:03 octocore python3[471]: split_referer = referer.split(":", 1) Mar 26 04:10:03 octocore python3[471]: AttributeError: 'NoneType' object has no attribute 'split' Mar 26 04:10:03 octocore python3[471]: ---------------------------------------- Mar 26 04:10:03 octocore python3[471]: Exception happened during processing of request from ('192.168.88.1', 40405) Mar 26 04:10:03 octocore python3[471]: Traceback (most recent call last): Mar 26 04:10:03 octocore python3[471]: File "/usr/lib/python3.7/socketserver.py", line 650, in process_request_thread Mar 26 04:10:03 octocore python3[471]: self.finish_request(request, client_address) Mar 26 09:38:44 octocore systemd[1]: startDuetLapse3.service: Current command vanished from the unit file, execution of the command list won't be resumed. pi@octocore:/etc/systemd/system $Looking at that line 198 in startDuetLapse3.py it looks like it has something to do with parsing the URL.
Here is the ExecStart line from the service file:
ExecStart=python3 /opt/DuetLapse/startDuetLapse3.py -port 8082 -args="-duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect layer -extratime 4"Thoughts?
John
