DuetLapse3

stuartofmt

I'll go back to the DWC threads and see if there are some ideas I can try.

The good news is I've made the baseline code considerable more robust and have improved the handling of pauses (if used). But none of the pause improvements matter without the ability to send gcodes

PCR

@stuartofmt
https://github.com/Kriechi/Cura-DuetRRFPlugin/blob/master/DuetRRFOutputDevice.py

Maybe this is helpful

PCR

@stuartofmt only for history

pi@duet3:~ $ ./1610998509032-sbctest.py -duet 192.168.140.126
192.168.140.126 Determine API Version
192.168.140.126 Testing: rr_model at address 192.168.140.126
192.168.140.126 http://192.168.140.126/rr_model?key=boards
192.168.140.126
192.168.140.126 Testing: /machine/system at address 192.168.140.126
192.168.140.126 http://192.168.140.126/machine/status
192.168.140.126
192.168.140.126 API model is: SBC   Version is: 3
192.168.140.126 Status is :processing
192.168.140.126 URL was: http://192.168.140.126/machine/code
192.168.140.126 Send Gcode result: None
192.168.140.126 Layer: 3
192.168.140.126 Get Position X and Y: 96.787 119.675
192.168.140.126 URL was: http://192.168.140.126/machine/code/
192.168.140.126 Send Gcode result: None

no DWC message either

PCR

@stuartofmt I changed this line

logger.info('Send Gcode result: '+str(sendDuetGcode2(apiModel, 'M117 Test of SBC')))

to

logger.info('Send Gcode result: '+str(sendDuetGcode2(apiModel, 'M117 "Test of SBC"')))

aand

stuartofmt

@PCR said in DuetLapse3:

logger.info('Send Gcode result: '+str(sendDuetGcode2(apiModel, 'M117 "Test of SBC"')))
aand

![a02d93f9-98a1-4404-bb9a-c67b729cac68-grafik.png](/assets/uploads/files/1611049373109-a02d93f9-98a1-4404-bb9a-c67b729cac68-grafik.png)

Brilliant!.
Last night (in my slumbers) I wonderer if the data=command in the push code needed to be encoded because of the spaces in M117 etc. In the various example code it seems only single gcodes have been passed and likely why those examples worked. Dueui was a bit different but that was javascript and I'm even more unfamiliar with that than I am with Python .....
DuetLapse3 in its current form only needs to send single gcodes. So - as it stands we have a fully working DuetLapse3. The choice of M117 ... for testing turned out to be both good and bad.

What I will do today is try and crack the encoding needs so that it does not break in the future. I've set up a simple webserver and can debug the incoming and see what happens. I have not yet updated the github but either way will do so later today

@jay_s_uk and @PCR
Thanks for being the remote testers.

stuartofmt

Version 3.0.4 released in github. All functionality should now work with SBC.

The issues with SBC gcode appear to be due to differences in the way DWS was handling the test case.

stuartofmt

Version 3.1.1 released on github. This is pretty significant - the details are documented but here us a summary.

###Version 3.1.0###

[1] Added support for Windows
[2] Added automatic detection of operating system (Linux or Windows)
[3] Added integrated http listener for basic browser based control
[4] Changed file naming convention to make filenames unique if multiple instances of DuetLapse3 are running.

An Integrated http listener is available for basic control of DuetLapse3 (not the attached printer).
The http listener is activated by specifying a port number using the -port option.
In conjunction with the -host option it provides the following functionality from a browser (or curl or other method of performing a http get).

http://<ip-address><port>/?command=<valid command>

*Valid commands are:
start - Starts DuetLapse3 recording if the -stopcmd option was used or after a stop command
standby - Stops (but does not terminate) DuetLapse3 recording and discards any images capture. Waits for a start command.
status - returns brief information about the running state of DuetLapse3
pause - causes DuetLapse3 to temporarily stop capturing images
continue - causes DuetLapse3 to resume capturing images
snapshot - causes DuetLapse3 to make an interim video and then continue
restart - causes DuetLapse3 to stop capturing images, create a video and then restart with a new capture set
terminate - causes DuetLapse3 to stop capturing images, create a video and then terminate the program. This is the same as CTRL+C or SIGINT.

Note that depending on your system - it may take several minutes for the http listener to completely shutdown following a terminate request.*

Note that the http listener will stop responding if DuetLapse3 is run from a command console that is then closed. This will happen even if started in background. To avoid this - use nohup (linux).
Windows does not have an (easy) equivalent to nohup so you would need to leave the command console open. An alternative if you are on Win10 is to use Windows Subsystem for Linux (WSL) and run DuetLapse as a linux application inside WSL.
If running in nohup mode CTRL+C cannot be used so you need to send the terminate command (?command=terminate) from the http listener

Made an accommodation (in 3.1.1) for ffmpeg < 4.2 by using a simpler syntax if -extratime is not used.

stuartofmt

@fractalengineer

Take a look at the post above. It does not provide a plugin but the http listener makes it easier to control remotely. It looks like the plugins require javascript and frankly - that's not something I want to get into.
Someone else (with the skills) could create a simple plugin that calls appropriate urls and provide control that way.
I will create a helper script that proved a separate http listener to remotely start DuetLapse3 - but that's a few days away.

fractalengineer

@stuartofmt Thanks Stuart; I did try installing the 3.04 earlier this weekend following each step but got stuck at ffmpeg install; doesn't return anything when checking version.

I guess I'm still way to beginner at linux terminal. I'll try again with the new version

stuartofmt

@fractalengineer said in DuetLapse3:

@stuartofmt Thanks Stuart; I did try installing the 3.04 earlier this weekend following each step but got stuck at ffmpeg install; doesn't return anything when checking version.

I guess I'm still way to beginner at linux terminal. I'll try again with the new version

Don't worry - we will get you going.
I'm assuming you are using a raspberry pi ?

The latest version should work with the version of ffmpeg that installs by default with the pi. The only downside is not being able to use the -extratime option.

I'd recommend you reinstall the default ffmpeg and take it from there.
It sounds like you tried to follow my instructions and the installation of the latest ffmpeg failed somehow? If that is correct then lets try to clean it up and get you back to the default. Try the following in order (I just did this on my pi)

cd ~
sudo apt remove ffmpeg

I'm guessing this will say ffmpeg is not installed.

sudo find / -name ffmpeg

This will may give some output for example:)
find: ‘/run/user/1000/gvfs’: Permission denied
/usr/local/bin/ffmpeg
/usr/local/share/ffmpeg
/usr/local/share/doc/ffmpeg

for each of the files / directories - we'll first delete everything for EXAMPLE

sudo rm -rf /usr/local/bin/ffmpeg
sudo rm -rf /usr/local/share/ffmpeg
sudo rm -rf /usr/local/share/doc/ffmpeg

Next make sure everything is at the latest release level

sudo apt update
sudo apt upgrade

I just did this on my pi - it was quite the upgrade and took quite a while so be patient.

Finally, install the standard ffmpeg

sudo apt install ffmpeg

after that and assuming no errors

ffmpeg -version

should return a bunch of information.

Then reboot for luck

fractalengineer

@stuartofmt aww mate you're a saint; didn't expect such a detailed help.

I went through those steps and when doing the find it didn't return anything; so I guess it didn't build properly in the first place (I used the cmake last time)

Seems like everything is installed now; just got to figure out the usage exactly

Thanks!

SIam

Hi,

@stuartofmt I have a Problem with layer capturing if i use this nothing will be captured i use the following command to start it

python3 DuetLapse3.py -port 8181 -duet 192.168.188.36 -detect layer -camera1 stream -weburl1 http://192.168.188.34:8081/

This result in

192.168.188.36 ################### Options selected for this run #####################
192.168.188.36 #Environment Settings:
192.168.188.36 # printer         = 192.168.188.36
192.168.188.36 # basedir         = /home/pi/DuetLapse
192.168.188.36 # poll            = 5
192.168.188.36 # logtype         = both
192.168.188.36 # verbose         = False
192.168.188.36 # os              = Linux
192.168.188.36 # host            = 0.0.0.0
192.168.188.36 # port            = 8181
192.168.188.36 # pid            = 7902
192.168.188.36 #Execution Setings:
192.168.188.36 # dontwait        = False
192.168.188.36 # seconds         = 0
192.168.188.36 # detect          = layer
192.168.188.36 # pause           = no
192.168.188.36 # standby     = False
192.168.188.36 #Camera1 Settings:
192.168.188.36 # camera1         = stream
192.168.188.36 # weburl1         = http://192.168.188.34:8081/
192.168.188.36 # Video Settings:
192.168.188.36 # extratime   =     0
192.168.188.36 ###################################################################
192.168.188.36
192.168.188.36
192.168.188.36
192.168.188.36 ###############################################################
192.168.188.36 Connected to printer at 192.168.188.36 using version 3.2 and API access using rr_model
192.168.188.36 ###############################################################
192.168.188.36
192.168.188.36
192.168.188.36 ##########################################################
192.168.188.36 ***** Started http listener *****
192.168.188.36 ##########################################################
192.168.188.36
192.168.188.36
192.168.188.36 ##########################################################
192.168.188.36 Will start capturing images on first layer change
192.168.188.36 ##########################################################
192.168.188.36
192.168.188.36
192.168.188.36 ##########################################################
192.168.188.36 Video will be created when printing ends.
192.168.188.36 Or, press Ctrl+C one time to stop capture and create video.
192.168.188.36 ##########################################################
192.168.188.36
192.168.188.36 ****** Printer State changed to waiting *****
192.168.188.36 ****** Printer State changed to printing *****
192.168.188.36 ****** Printer State changed to completed *****
192.168.188.36 End of Print Job - Post Processing
192.168.188.36 ++++++ Entering restart state ++++++
192.168.188.36
192.168.188.36 Camera1: Cannot create video with only 0 frames
192.168.188.36 ****** Printer State changed to waiting *****

If i use the Time capturing option it works as it should

/DuetLapse $ python3 DuetLapse3.py -port 8181 -duet 192.168.188.36 -seconds 20 -detect none -camera1 stream -weburl1 http://192.168.188.34:8081/
..
..
..
192.168.188.36 ##########################################################
192.168.188.36 Video will be created when printing ends.
192.168.188.36 Or, press Ctrl+C one time to stop capture and create video.
192.168.188.36 ##########################################################
192.168.188.36
192.168.188.36 ****** Printer State changed to waiting *****
192.168.188.36 ****** Printer State changed to printing *****
192.168.188.36 ****** Printer State changed to startnow *****
192.168.188.36 Camera1: capturing frame 0 at layer 72 after 20.0 seconds
192.168.188.36 ****** Printer State changed to printing *****
192.168.188.36 Camera1: capturing frame 1 at layer 73 after 20.0 seconds
192.168.188.36 Camera1: capturing frame 2 at layer 74 after 20.0 seconds
192.168.188.36 Camera1: capturing frame 3 at layer 75 after 20.0 seconds
192.168.188.36 Camera1: capturing frame 4 at layer 76 after 20.0 seconds
192.168.188.36 Camera1: capturing frame 5 at layer 78 after 20.0 seconds
192.168.188.36 Camera1: capturing frame 6 at layer 79 after 20.0 seconds
192.168.188.36 Camera1: capturing frame 7 at layer 80 after 20.0 seconds
^C192.168.188.36 !!!!!! Stopped by SIGINT - Post Processing !!!!!!
192.168.188.36
192.168.188.36 Camera1: Cannot create video with only 8 frames
192.168.188.36 Waiting for http listener to shutdown.
192.168.188.36 Program Terminated

The printer is on the latest stable:

RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2 running on Duet WiFi 1.02 or later + DueX5

Can you please tell me what i did wrong?

SIam

my knowledge of python is not really available but when i look at this code:

 #update the layer counter
    global zo1, zo2
    zn = getDuetLayer(apiModel)
    if (cameraname == 'Camera1'):
        zo1 = zn
    else:
        zo2 = zn
    
    if ('layer' in detect):
        if ((not zn == zo1 and cameraname == 'Camera1') or (not zn == zo2 and cameraname == 'Camera2')):
            # Layer changed, take a picture.
            checkForPause()
            logger.info(cameraname+': capturing frame '+str(frame)+' at layer '+str(zn)+' after layer change')           
            onePhoto(cameraname, camera, weburl, camparam)            

    elif ('pause' in detect and printerState == 'paused'):
            checkForPause()
            logger.info(cameraname+': capturing frame '+str(frame)+' at layer '+str(zn)+' at pause in print gcode')
            onePhoto(cameraname, camera, weburl, camparam)

So this script can't access to the take picture routine because zo1 and zn is always the same

I have now altered the code to:

    #update the layer counter
    global zo1, zo2
    zn = getDuetLayer(apiModel)
   # if (cameraname == 'Camera1'):
   #     zo1 = zn
   # else:
   #     zo2 = zn

    if ('layer' in detect):
        if ((not zn == zo1 and cameraname == 'Camera1') or (not zn == zo2 and cameraname == 'Camera2')):
            # Layer changed, take a picture.
            checkForPause()
            logger.info(cameraname+': capturing frame '+str(frame)+' at layer '+str(zn)+' after layer change')
            onePhoto(cameraname, camera, weburl, camparam)

    elif ('pause' in detect and printerState == 'paused'):
            checkForPause()
            logger.info(cameraname+': capturing frame '+str(frame)+' at layer '+str(zn)+' at pause in print gcode')
            onePhoto(cameraname, camera, weburl, camparam)

   # global zo1, zo2
   # zn = getDuetLayer(apiModel)
    if (cameraname == 'Camera1'):
        zo1 = zn
    else:
        zo2 = zn

And for now it works (but i don't know if i have now added a new failure)

stuartofmt

@SIam

Hi Sam - sorry you are having problems. The changes you made to the code will most definitely cause problems That piece of code keeps track of layer changes.

Having said that it was my error ...

But first - I see you have started DuetLapse with:

python3 DuetLapse3.py -port 8181 -duet 192.168.188.36 -detect layer -camera1 stream -weburl1 http://192.168.188.34:8081/

You have BOTH your web camera (-weburl1) AND DuetLapse3 on the same port number. This is a no-no. They need to be different. In the next release (I am testing now) - I check to see if the port is already in use and if it is, prevent DuetLapse3 from starting. So please change the -port so that it is on a different and unused port number.

Second - on the layer change - yikes! This is definitely wrong and thanks for pointing it out. I must have copy/pasted that portion into the wrong place when I did some 'tidy up' . I will fix that today. If you want to fix it yourself - move the assignment so that it looks like this:

    #update the layer counter
    global zo1, zo2
    zn = getDuetLayer(apiModel)

    
    if ('layer' in detect):
        if ((not zn == zo1 and cameraname == 'Camera1') or (not zn == zo2 and cameraname == 'Camera2')):
            # Layer changed, take a picture.
            checkForPause()
            logger.info(cameraname+': capturing frame '+str(frame)+' at layer '+str(zn)+' after layer change')           
            onePhoto(cameraname, camera, weburl, camparam)            

    elif ('pause' in detect and duetStatus == 'paused'):
            checkForPause()
            logger.info(cameraname+': capturing frame '+str(frame)+' at layer '+str(zn)+' at pause in print gcode')
            onePhoto(cameraname, camera, weburl, camparam)
            
    if (cameraname == 'Camera1'):
        zo1 = zn
    else:
        zo2 = zn

In any case, the change to -seconds 20 -detect none is working as it should but that is bypassing the layer check so the change you made has no effect.

SIam

Thank you for your assistance!

My start command is correct duetlapse runs on 8181 and the cam on 8081

to your code change the only change i can see is that you have removed the comments is this right ?

stuartofmt

@SIam said in DuetLapse3:

Thank you for your assistance!

My start command is correct duetlapse runs on 8181 and the cam on 8081

to your code change the only change i can see is that you have removed the comments is this right ?

Sorry - I misread the port numbers , you are of course correct.

As to the question on the code change - no.

This portion of code

    if (cameraname == 'Camera1'):
        zo1 = zn
    else:
        zo2 = zn

needs to be after this portion (not before - which was the problem )

    if ('layer' in detect):
        if ((not zn == zo1 and cameraname == 'Camera1') or (not zn == zo2 and cameraname == 'Camera2')):
            # Layer changed, take a picture.
            checkForPause()
            logger.info(cameraname+': capturing frame '+str(frame)+' at layer '+str(zn)+' after layer change')           
            onePhoto(cameraname, camera, weburl, camparam)            
 
    elif ('pause' in detect and duetStatus == 'paused'):
            checkForPause()
            logger.info(cameraname+': capturing frame '+str(frame)+' at layer '+str(zn)+' at pause in print gcode')
            onePhoto(cameraname, camera, weburl, camparam)

NOTE THAT THE INDENTING HAS TO BE EXACTLY THE SAME - Python is very particular in that regard - so be careful as you edit

SIam

ok! Thank you for your work i love it

SIam

is it possible to catch those errors in python ?

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/DuetLapse/DuetLapse3.py", line 964, in <module>
    duetStatus=getDuetStatus(apiModel)
  File "/home/pi/DuetLapse/DuetLapse3.py", line 694, in getDuetStatus
    r = requests.get(URL, timeout=120)  #Long timeout to handle restart of Duet
  File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
^C
pi@raspberrypi:~/DuetLapse $ ./start-lapse stop
192.168.188.36 !!!!!! Stopped by SIGINT - Post Processing !!!!!!
192.168.188.36
pi@raspberrypi:~/DuetLapse $ 192.168.188.36 Camera1: now making 29 frames into a video
192.168.188.36 Video processing complete for Camera1
192.168.188.36 Video is in file "/home/pi/DuetLapse/192-168-188-36/Camera17144-Mon-11-25.mp4"
192.168.188.36 Waiting for http listener to shutdown.
192.168.188.36 Program Terminated
Exception ignored in: <module 'threading' from '/usr/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 1281, in _shutdown
    t.join()
  File "/usr/lib/python3.7/threading.py", line 1032, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
  File "/home/pi/DuetLapse/DuetLapse3.py", line 467, in quit_gracefully
    terminate()
  File "/home/pi/DuetLapse/DuetLapse3.py", line 798, in terminate
    sys.exit(0)
SystemExit: 0

stuartofmt

Version 3.2.0 has been release.
This is a significant upgrade and other than bug fixes I expect it will be stable for a while.

I added a helper program startDuetLapse3 to enable remote start/stop of DuetLapse3 instances using a browser or similar (e.g. curl). This provides a lot of flexibility in that you can have the helper program set to run at boot time and then remotely start / stop DuetLapse3 instances when you need them without having to use additional scripts or SSH into the machine etc.

###Version 3.2.0###

[1] Adds the ability to work with startDuetLapse3. See the description here:
https://github.com/stuartofmt/DuetLapse3/blob/main/startDuetLapse3.md
In summary startDuetLapse3 provides the ability to remotely start / stop DuetLapse3.
[2] Changed the shutdown method to reduce low level 'noise' messages.
[3] Restructured to provide better efficiency. Typically less than 1% CPU when idle.
[4] Fixed a bug that was introduced in the prior version (not detecting layer changes)
[5] Made further improvements to the handling of pauses.
There are some edge cases where a printer can get stuck in a paused state.
The documentation has been updated. See especially -dontwait
[6]Added checks to prevent startup with ports that are already in use.

I have done a lot of testing but as usual - something may have slipped through the cracks .....

stuartofmt

@SIam said in DuetLapse3:

is it possible to catch those errors in python ?

Yes - it was one of the things that annoyed me too! one of those things that a 1000 google 'fixes' (all different) did not fix I found a way though
Its part of the 3.2.0 release.

Hope I got them all