DuetLapse3

[SIam]

ok! Thank you for your work i love it

[SIam]

is it possible to catch those errors in python ?

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/DuetLapse/DuetLapse3.py", line 964, in <module>
    duetStatus=getDuetStatus(apiModel)
  File "/home/pi/DuetLapse/DuetLapse3.py", line 694, in getDuetStatus
    r = requests.get(URL, timeout=120)  #Long timeout to handle restart of Duet
  File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
^C
pi@raspberrypi:~/DuetLapse $ ./start-lapse stop
192.168.188.36 !!!!!! Stopped by SIGINT - Post Processing !!!!!!
192.168.188.36
pi@raspberrypi:~/DuetLapse $ 192.168.188.36 Camera1: now making 29 frames into a video
192.168.188.36 Video processing complete for Camera1
192.168.188.36 Video is in file "/home/pi/DuetLapse/192-168-188-36/Camera17144-Mon-11-25.mp4"
192.168.188.36 Waiting for http listener to shutdown.
192.168.188.36 Program Terminated
Exception ignored in: <module 'threading' from '/usr/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 1281, in _shutdown
    t.join()
  File "/usr/lib/python3.7/threading.py", line 1032, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
  File "/home/pi/DuetLapse/DuetLapse3.py", line 467, in quit_gracefully
    terminate()
  File "/home/pi/DuetLapse/DuetLapse3.py", line 798, in terminate
    sys.exit(0)
SystemExit: 0

[stuartofmt]

Version 3.2.0 has been release.
This is a significant upgrade and other than bug fixes I expect it will be stable for a while.

I added a helper program startDuetLapse3 to enable remote start/stop of DuetLapse3 instances using a browser or similar (e.g. curl). This provides a lot of flexibility in that you can have the helper program set to run at boot time and then remotely start / stop DuetLapse3 instances when you need them without having to use additional scripts or SSH into the machine etc.

###Version 3.2.0###

[1] Adds the ability to work with startDuetLapse3. See the description here:
https://github.com/stuartofmt/DuetLapse3/blob/main/startDuetLapse3.md
In summary startDuetLapse3 provides the ability to remotely start / stop DuetLapse3.
[2] Changed the shutdown method to reduce low level 'noise' messages.
[3] Restructured to provide better efficiency. Typically less than 1% CPU when idle.
[4] Fixed a bug that was introduced in the prior version (not detecting layer changes)
[5] Made further improvements to the handling of pauses.
There are some edge cases where a printer can get stuck in a paused state.
The documentation has been updated. See especially -dontwait
[6]Added checks to prevent startup with ports that are already in use.

I have done a lot of testing but as usual - something may have slipped through the cracks .....

[stuartofmt]

@SIam said in DuetLapse3:

is it possible to catch those errors in python ?

Yes - it was one of the things that annoyed me too! one of those things that a 1000 google 'fixes' (all different) did not fix I found a way though
Its part of the 3.2.0 release.

Hope I got them all

[JohnOCFII]

@stuartofmt I'm looking forward to testing this and eliminating my auto-start setup. I find my older setup is re-starting multiple times a day, and only noticed it doing longer prints, when the timelapse was only for the last period of time. Kind of sad after a 10 hour print yielded a timelapse of the last 4 minutes...

[stuartofmt]

@JohnOCFII said in DuetLapse3:

..... Kind of sad after a 10 hour print yielded a timelapse of the last 4 minutes...

I hear ya! FWIW - I have "soak tested" this running overnight and into the next day with a small number of seconds (smaller than practical). 10K+ images and it was stable but I'm sure you can find something for me to fix

Note that it also runs on WIN10 (and maybe earlier ??) as well as WSL (which I like a lot). So if the Pi becomes a limitation there are other options.

Let me know how it works out.

[SIam]

hi,

thanks for your work!! Now i have a question for what is the whitelist ? i get this error:

Traceback (most recent call last):
  File "DuetLapse3.py", line 18, in <module>
    from DuetLapse3 import whitelist
  File "/home/pi/DuetLapse/DuetLapse3.py", line 18, in <module>
    from DuetLapse3 import whitelist
ImportError: cannot import name 'whitelist' from 'DuetLapse3' (/home/pi/DuetLapse/DuetLapse3.py)

the module is installed

pi@raspberrypi:~/DuetLapse $ sudo find / -name whitelist*
/home/pi/.local/lib/python3.7/site-packages/whitelist
/home/pi/.local/lib/python3.7/site-packages/whitelist-0.2.8.dist-info

[stuartofmt]

@SIam said in DuetLapse3:

hi,

thanks for your work!! Now i have a question for what is the whitelist ? i get this error:

The short answer is that startDuetLapse.py needs to be in the same directory as DuetLapse3.py AND DuetLapse3.py needs to be at least at V3.2.0 (i.e. the latest version) .

I have never seen this error - but then I satisfy the above :-).

I just tried to simulate it by putting startDuetLapse3.py in a different directory and get a related but slightly different error. So I suspect that you may have an older version of DuetLapse3.py ?

python3 ./DuetLapse3.py -h

Needs to have this line in its return (up near the top)
Create time lapse video for Duet3D based printer. V3.2.0

The longer answer is this: When DuetLapse3.py is started manually, the various input options are checked for validity (i.e. parsed through a whitelist).
To help prevent misuse and provide a measure of security startDuetLapse3.py invokes the very same code (imported from DuetLapse3.py) BEFORE it tries to start DuetLapse3.py (which then dutifully does the same thing again).

Why bother apparently doing it twice? To help prevent "someone" (not you, someone malicious) from accessing startDuetLapse3 via the http port and trying to execute some arbitrary code.
The original version (proof of concept) just parsed through any old command line. Dangerous indeed.
To feel comfortable - I needed startDuetLapse3 to be very specific to DuetLapse3. One of those defenses (minimal but likely 'good enough') was to use a whitelist. Since it did not make sense to try and maintain the whitelist it in two places - I restructured some of the code in DuetLapse3 3.2.0 to allow the whitelist to be imported. Hence the dependency on versions and location.

[SIam]

Sorry this was my fault my SD card was full after cleanup the SD card and Reinstallation the duet lapse is working.

[jnofsinger]

Hello,

Trying out 3.2.0 and running into the following issue (below). The print starts fine and captures a handful of images till the connection reset issue. Both the duet and the pi are running on the same wifi subnet, and the duet is the latest firmware version.

running with the following options (have more logs if needed)

/usr/bin/python3 /home/pi/Desktop/images/DuetLapse3.py -port 8181 -verbose -duet 172.16.37.118 -detect layer -camera1 stream -weburl1 http://127.0.0.1:8080/?action=stream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/home/pi/Desktop/images/DuetLapse3.py", line 1068, in captureLoop
oneInterval('Camera1', camera1, weburl1, camparam1)
File "/home/pi/Desktop/images/DuetLapse3.py", line 612, in oneInterval
zn = getDuetLayer(apiModel)
File "/home/pi/Desktop/images/DuetLapse3.py", line 733, in getDuetLayer
r = requests.get(URL, timeout=5)
File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection abort

[SIam]

@stuartofmt sorry but i have also a error for you

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer                                                                 '))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pi/DuetLapse/DuetLapse3.py", line 1041, in captureLoop
    duetStatus=getDuetStatus(apiModel)
  File "/home/pi/DuetLapse/DuetLapse3.py", line 707, in getDuetStatus
    r = requests.get(URL, timeout=120)  #Long timeout to handle restart of Duet
  File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by p                                                                 eer'))

[stuartofmt]

@jnofsinger said in DuetLapse3:

File "/home/pi/Desktop/images/DuetLapse3.py", line 733, in getDuetLayer
r = requests.get(URL, timeout=5)

Your command is almost identical to one I used successfully yesterday - the only meaningful difference being -verbose.

The problem does not look like there was a timeout issue - if there was there should be a message in the logs like:

getDuetLayer failed to get data. Code: <some number> Reason: <some text>

SO that points to a communication issue between the pi and the Duet i.e. it looks like it could not even connect. The fact that it worked sometimes makes it hard to track down and suggests that the basic code is ok.

Let me thing about how to trap the connection issue and report it - although to be frank, I don't want to disappear down the rabbit hole of debugging Duet connections

In any case, in parallel, can you trawl through the WDC forum and see what discussions are taking place on connection issues. My sense is that you will need to address some aspect anyway , independently of anything I can do in the code.

I hope to have a patched version for you to try later today / tomorrow but as I said , I doubt it will fix anything - maybe just allow Duet to report the error or try a little harder.

[stuartofmt]

@SIam :

@jnofsinger

It looks like you are both experiencing the same issue. Are you both using SBC ?

Please try this version. I restructured the API calls and put in some exception handling for network connection issues and for timeouts. The messages will appear in the output. The code will try to re-establish a connection and continue on if it is successful.
It will eventually give up after a few minutes.
In any case it will give some clues as to what is happening.
I tested this in three ways (1) turning off the network on the machine running DuetLapse3 and (2) turning off my printer (3) repeatedly hitting emergency stop in DWC

This will show as version 3.2.1. Please report back.

[DuetLapse3.py]

[jnofsinger]

@stuartofmt

I have a Duet Wifi 1.01 board. it has been with me for a very long time and is impossible to kill ;).

I can just have the program open and it will throw those errors even when not printing. If I switch it to take picture every x seconds it will work fine. Your idea of some type of communication issue between the pi and duet is correct. I also have the webcam from the duet pointing to the same URL for a quick print and it stays connected.

I will try it again with the .1 version and see if it improves. Also I will remove the webcam url from the duet itself to see if something differs.

Thanks
Jim

[stuartofmt]

@jnofsinger said in DuetLapse3:

If I switch it to take picture every x seconds it will work fine.

Hmm - I'd love it if that was a clue but from a codepoint of view it asks Duet to report the current layer even if its just capturing images every 'nnn' seconds. So in theory* - that should not make any difference. But we shall see .....

• --> As I often say: "There is no difference between theory and practice -- except in practise"

[jnofsinger]

@stuartofmt

So far so good! I have a 72 frame capture and will print another test. The logs are clean and did not show any disconnects. Will continue to test it tonight and let you know.

Great program for the duet. I do have a question, do you plan on offering a flag to flip images with the ffmpeg processing? My camera is flipped as it comes from the top of my Ender3.

Thanks,
Jim

[stuartofmt]

@jnofsinger said in DuetLapse3:

@stuartofmt
do you plan on offering a flag to flip images with the ffmpeg processing? My camera is flipped as it comes from the top of my Ender3.

Thanks,
Jim

Jim - I do not plan to do this. I have spent some time trying and failed. The reason is that there are just too many versions of ffmpeg out there and finding a command sequence that actually works across many of them is a (time-consuming) art-form. For example: using "rotate" may or may not work on playback, using "transpose" does not work on the version of ffmpeg that I have .

In any case - I believe a better solution is to rotate at the source. That way you can also monitor the print the right way up
The program you are using to stream the video should (hopefully) have the ability to set the port and also to rotate the images before streaming. If not - you may be able to find another streaming program for your setup.

If you are using a Pi and a Pi camera -- here is what I use.
http://picamera.readthedocs.io/en/latest/recipes2.html#web-streaming

So far so good! I have a 72 frame capture and will print another test. The logs are clean and did not show any disconnects. Will continue to test it tonight and let you know.

Just to be clear - I did not do anything to fix the connection problem. I just try to ignore it and hope it will go away. If there are connection problems - these will be reported in the console / log. If you are not seeing these messages "There was a network failure:..." followed by several lines of explanation -- then the previous issue has mysteriously gone away (at least for now)

Good Luck.

[stuartofmt]

A small update to reflect the recent test code on handling Duet connectivity issues plus a couple of cosmetic tweaks.

###Version 3.2.2###

[1] Added resilience to lost connectivity with Duet. If disconnected the issue will be reported and attempts to reconnect will be made
[2] Made some cosmetic changes to the http responses. Most now include the local time as part of the response.
[3] The status page will automatically refresh every 60 seconds. Other pages will show the last time they were invoked.
[4]Similar changes to [2] and [3] made to startDuetLapse

[stuartofmt]

Another small update. For those using windows (c'mon I know you are out there !). There were some inconsistencies in the way python3 provided some operating system information.

###Version 3.2.3###

[1] Fixed some inconsistencies when running on Windows due to slightly different behavior of python3.

[GoremanX]

This post is deleted!