AI Print Monitoring / Printer Health monitoring

Peter Lebiedzinski

Introduction
Hello all, we are looking to gauge interest and get feedback on an AI integration for RRF.

Features

Visual Failure monitoring: uses frames from the webcam/IP camera/RTSP camera and detects spaghetti failure and notifies the users OR pauses/takes action on behalf of the printer.
Anomaly Detection: using ObjectModel data from the printer, monitors and tracks the current health of the printer and it's subcomponents. Used for scheduling preventative maintenance and detecting+diagnoising issues with the printer.

Comments
If you are interested in a feature/plugin/integration that provides you access with these AI features, please like or comment on this thread.

If you have requests for features that aren't listed and fall under the scope of AI + Data, please leave a comment or reach out via direct message/email.

Contact
Email: Lebiedzinskip@printpal.io

T3P3Tony

@Peter-Lebiedzinski thanks for posting here! I am looking forward to trying this out!

It could help to specify the minimum requirements for your implementation that people need to try it out.

Peter Lebiedzinski

@T3P3Tony

Of course, thank you for reminding me.

The minimum requirements are as follows:

Visual Failure Monitoring:

Firmware Requirements:
- RepRapFirmware 3.4.5b3+ and above, this firmware contains support for the application sending commands to the printer when a failure has occurred. The software will still work and be able to notify you of failures if not using this firmware, however no actions will be able to be taken on behalf of the printer.
Supported formats:
- HTTP/S or MJPEG stream (preferred)
- RTSP
- USB
- PiCamera
Resolution:
- 480 x 480 pixels @ 100% quality (no compression) --> 768 x 768 @ 100% quality or greater recommended
Hardware:
- Able to process 1 image every 10 seconds
- Internet connected (Ethernet or WiFi)
- Can communicate with the Duet3D board
- Examples:
  - Raspberry Pi 3
  - Raspberry Pi 4
  - Raspberry Pi zero W
  - ESP32 - does not have ethernet option to communicate with Duet3D board
- Edge option:
  - Embedded AI is an option and the hardware will fit inside the electronics cabinet of the printer. For this you'd have to contact us.

Anomaly Detection:

Firmware requirements:
- Most recent firmware with support for datalogging (TBD)
Hardware:
- Although this should be available for everyone in some capacity, hardware that provides more feedback information is better, this includes, but is not limited to:
  - Closed-loop steppers/servos
  - Accelerometer
  - Lidar
  - Temperature sensors
  - Flow sensors
  - Voltage/Amperage sensors

Peter Lebiedzinski

To all who upvoted (and those who didn't but are still interested): we have the working beta version of the AI monitoring plugin available for testing. Currently, the plugin will only work on Duet printers running in SBC mode.

The ZIP install file for the plugin may be found on GitHub: https://github.com/printpal-io/DSF_PrintWatchAI_Plugin

Please reach out to me personally (lebiedzinskip@printpal.io) to receive a free API key for testing without limits. The free version without the API key will only allow you up to 12 hours of AI monitoring.

This is what the plugin currently looks like, any feedback is taken seriously, and very much appreciated:

DNvM84

@Peter-Lebiedzinski
Hi Peter, I received your email.
Any recommendations as to where the camera should be placed?
I ask because while I was researching other versions of monitoring software, they stated that the camera should be looking at the table at no more than a specified angle above the build surface.

Thank you for keeping me in the loop.
DNvM

Peter Lebiedzinski

@DNvM84 I would recommend placing the camera off of any corner of the print bed, with the center of the frame pointed at where the majority of the print will take.

I would also recommend placing it above the print bed such that it is 15 - 45 degrees from the vector of the plane that defines the print bed:

The angle can be determined by what you are printing and by how it works with the toolhead on your machine. Play around and find out what works best. Also, if printing an item that has a highly detailed infill that may confuse the AI, you would want to have the camera at a lower/higher angle so it does not perceive the infill as highly dense details.

Gianluca

Hi Peter,
I am very interested in your work and would like to try it.
Unfortunately I can't install your plugin on my system.
I have downloaded the file "PrintWatch-3.4.2.zip" ver.1.0 but there are compatibility problems, with the latest firmware version 3.5.0.RC1.

Are there any suggestions for a successful installation, or will other versions be released?

Peter Lebiedzinski

@Gianluca I will create a ZIP installer for your version of DSF. I will get back to you within the hour.

Gianluca

@Peter-Lebiedzinski said in AI Print Monitoring / Printer Health monitoring:

@Gianluca I will create a ZIP installer for your version of DSF. I will get back to you within the hour.

Thank You!!!

CNCModeller

Sounds like a really interesting service.

I think it'd be a good investment in terms of the subscription when I'm in production mode, and the free license would be possibly enough for now and then small hobby printers.

https://printpal.io/printwatch/

Definitely going to keep up with developments and give it a try on my printers once they're back together again after the current round of upgrades.

CNCModeller

@Peter-Lebiedzinski

In terms of AI capabilities I guess the ultimate goal would be to recognise any deviation from the STL or gcode file associated with the print.

Perhaps the plugin would build a representative visual model from the gcode and then alarm if the print head deviated...

Interested stuff...

Peter Lebiedzinski

@CNCModeller I cannot say too much, but we have been working with 3D Convolutions and Pointclouds of various CAD models, and there are very exciting things coming in the future.

CNCModeller

@Peter-Lebiedzinski

I guess another way would be to put tracking markers on the bed and print head then visually track those to confirm the printer is following the gcode...

deckingman

@Peter-Lebiedzinski I have a few questions regarding "anomaly detection". Assuming a print is showing signs of under extrusion, would your system detect and report that? If so, is it "intelligent" enough to diagnose the reason which could be say a partially blocked nozzle, print speed too high and exceeding the maximum volumetric flow rate of the hot end, temperature too low for the filament in use, motor current too low for the extruder in use, or any of the other reasons that can cause under extrusion? Also, with things such as bulging corners or other surface defects, is it intelligent enough to discriminate between genuine defects and features of the model itself?

Peter Lebiedzinski

@deckingman This would depend on what sensor values are available. If we have feedback data from the extruder motor (current, position, command position, etc), then yes. That is why we like the way Duet is moving with adding support for closed-loop steppers and low-level variable sampling at 4 Hz.

As for diagnosing the root cause of the issue, maybe, but not with high certainty, at least not as of now with our testing. We plan on trying various simple methods first before developing an end-to-end model for this, some of which would be clustering the data from our AD ("Anomaly Detection") model and seeing if we can segment out any distinct clusters that we can label as "pertaining to an issue with X, Y, or Z".

Regarding your last question, if it is a computer vision model then it is very unlikely to be able to discriminate genuine defects vs model features. It takes everything at face value as it sees it on the sensor. This of course could be addressed with a CAD model informed ML model, but we do not have plans on developing that as of now.

Peter Lebiedzinski

@CNCModeller This would be helpful, but not necessary. At the end of the day we want to make it as modular and easy to use for the user.

With nowadays computer vision operations, we can easily pointcloud the current print from a single camera angle.

We could perhaps wrap this into a feature rather quickly and release it for beta testing. The end feature would be a detector for clog, something conventional object detectors struggle to detect as it is trying to detect the absence of something in the frame.

Peter Lebiedzinski

@Gianluca
We have created a ZIP installer for the 3.5.0-rc.1 versions of RRF/DSF/DWC, it can be found here: https://github.com/printpal-io/DSF_PrintWatchAI_Plugin/releases/tag/v3.5.0-rc.1

Please note that the 3.5.0-rc.1 currently has an issue with the HTTP endpoints being registered. This can be easily worked around by doing the following:

Make sure the plugin is installed and started
SSH into the SBC
Enter the command: sudo systemctl restart duetwebserver

This will restart the webserver and register the endpoints correctly. This must be done every time the SBC is started up. This should be fixed in the next release.

Gianluca

@Peter-Lebiedzinski said in AI Print Monitoring / Printer Health monitoring:

@Gianluca
We have created a ZIP installer for the 3.5.0-rc.1 versions of RRF/DSF/DWC, it can be found here: https://github.com/printpal-io/DSF_PrintWatchAI_Plugin/releases/tag/v3.5.0-rc.1

Please note that the 3.5.0-rc.1 currently has an issue with the HTTP endpoints being registered. This can be easily worked around by doing the following:

Make sure the plugin is installed and started

SSH into the SBC

Enter the command: sudo systemctl restart duetwebserver

This will restart the webserver and register the endpoints correctly. This must be done every time the SBC is started up. This should be fixed in the next release.

Thank you so much for your precious help. This evening, after work, I will do all the tests and update you. I'm sure others who have installed the latest firmware version will find your work very useful!

Peter Lebiedzinski

@DNvM84 @jay_s_uk
I've created a Plugin for DWC-only configurations to be able to run the AI monitoring. It will work for DWC 3.4, 3.4.6, and 3.5.0-rc as of now. It can be found here: https://github.com/printpal-io/DWC_PrintWatchAI_Plugin

In order to use the plugin, you still need a device (Raspberry Pi, or any SBC) running the AI backend component, and it needs to be on the same network (Ethernet or WiFi) as the Duet mainboard. The backend component as well as how to set it up can be found here: https://github.com/printpal-io/PrintWatchAI_Backend

I would recommend using a Raspberry Pi for running the backend since they are simple and easy to use.
I have also tested the Plugin's Backend component with great success on an Orange Pi Zero3 board and it works just as well. The Orange Pi Zero3 is a little more economical ($22.99 on amazon: https://www.amazon.com/Orange-Pi-Allwinner-Quad-Core-Development/dp/B0CB1BYTT8/ref=sr_1_5?crid=BYJ6JUDXT64O&keywords=orange+pi+zero+3&qid=1694217352&sprefix=orange+pi+zero+3%2Caps%2C118&sr=8-5)

When testing, please run with Test Mode enabled, since the code that checks if the printer is 'Printing' has not been fully tested.

Gianluca

@Peter-Lebiedzinski
I first tried to install PrintWatchAI_Backend on my RPI 4 from the command prompt, and in point 5 of the list I get an error that prevents the installation. I attach a list.

pi@VORON-24-SBC:~ $ cd /home/pi
pi@VORON-24-SBC:~ $ wget https://github.com/printpal-io/PrintWatchAI_Backend/archive/refs/tags/raspberrypi.zip
--2023-09-21 11:40:21--  https://github.com/printpal-io/PrintWatchAI_Backend/archive/refs/tags/raspberrypi.zip
Risoluzione di github.com (github.com)... 140.82.121.4
Connessione a github.com (github.com)|140.82.121.4|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 302 Found
Posizione: https://codeload.github.com/printpal-io/PrintWatchAI_Backend/zip/refs/tags/raspberrypi [segue]
--2023-09-21 11:40:21--  https://codeload.github.com/printpal-io/PrintWatchAI_Backend/zip/refs/tags/raspberrypi
Risoluzione di codeload.github.com (codeload.github.com)... 140.82.121.10
Connessione a codeload.github.com (codeload.github.com)|140.82.121.10|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 200 OK
Lunghezza: non specificato [application/zip]
Salvataggio in: "raspberrypi.zip"

raspberrypi.zip         [ <=>                ]  19,14K  --.-KB/s    in 0,03s   

2023-09-21 11:40:22 (550 KB/s) - "raspberrypi.zip" salvato [19599]

pi@VORON-24-SBC:~ $ unzip raspberrypi.zip && rm raspberrypi.zip
Archive:  raspberrypi.zip
ff8985d86b20f8dac85b9908a5057b61b5a3df26
replace PrintWatchAI_Backend-raspberrypi/LICENSE? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
  inflating: PrintWatchAI_Backend-raspberrypi/LICENSE  
  inflating: PrintWatchAI_Backend-raspberrypi/README.md  
replace PrintWatchAI_Backend-raspberrypi/main.py? [y]es, [n]o, [A]ll, [N]one, [r]ename: a
error:  invalid response [a]
replace PrintWatchAI_Backend-raspberrypi/main.py? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
  inflating: PrintWatchAI_Backend-raspberrypi/main.py  
  inflating: PrintWatchAI_Backend-raspberrypi/printwatch.service  
 extracting: PrintWatchAI_Backend-raspberrypi/printwatch/__init__.py  
  inflating: PrintWatchAI_Backend-raspberrypi/printwatch/client.py  
  inflating: PrintWatchAI_Backend-raspberrypi/printwatch/core.py  
  inflating: PrintWatchAI_Backend-raspberrypi/printwatch/interface.py  
  inflating: PrintWatchAI_Backend-raspberrypi/printwatch/utils.py  
  inflating: PrintWatchAI_Backend-raspberrypi/requirements.txt  
  inflating: PrintWatchAI_Backend-raspberrypi/ustreamer.service  
pi@VORON-24-SBC:~ $ cd PrintWatchAI_Backend-raspberrypi
pi@VORON-24-SBC:~/PrintWatchAI_Backend-raspberrypi $ pip3 install -r requirements.txt
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from -r requirements.txt (line 1)) (40.8.0)
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (0.32.3)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 3)) (3.8.5)
Requirement already satisfied: ujson in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 4)) (5.7.0)
Collecting typing (from -r requirements.txt (line 5))
  Using cached https://www.piwheels.org/simple/typing/typing-3.7.4.3-py3-none-any.whl
Collecting uvicorn (from -r requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/ad/bd/d47ee02312640fcf26c7e1c807402d5c5eab468571153a94ec8f7ada0e46/uvicorn-0.22.0-py3-none-any.whl
Requirement already satisfied: Pillow in /usr/lib/python3/dist-packages (from -r requirements.txt (line 7)) (5.4.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from -r requirements.txt (line 8)) (2.21.0)
Collecting fastapi (from -r requirements.txt (line 9))
  Using cached https://files.pythonhosted.org/packages/76/e5/ca411b260caa4e72f9ac5482f331fe74fd4eb5b97aa74d1d2806ccf07e2c/fastapi-0.103.1-py3-none-any.whl
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (23.1.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (1.3.3)
Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (4.7.1)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (3.2.0)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (1.9.2)
Requirement already satisfied: asynctest==0.13.0; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (0.13.0)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (4.0.3)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->-r requirements.txt (line 3)) (6.0.4)
Requirement already satisfied: click>=7.0 in /usr/lib/python3/dist-packages (from uvicorn->-r requirements.txt (line 6)) (7.0)
Collecting h11>=0.8 (from uvicorn->-r requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/95/04/ff642e65ad6b90db43e668d70ffb6736436c7ce41fcc549f4e9472234127/h11-0.14.0-py3-none-any.whl
Collecting anyio<4.0.0,>=3.7.1 (from fastapi->-r requirements.txt (line 9))
  Using cached https://files.pythonhosted.org/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl
Collecting pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 (from fastapi->-r requirements.txt (line 9))
  Using cached https://files.pythonhosted.org/packages/82/06/fafdc75e48b248eff364b4249af4bcc6952225e8f20e8205820afc66e88e/pydantic-2.3.0-py3-none-any.whl
Collecting starlette<0.28.0,>=0.27.0 (from fastapi->-r requirements.txt (line 9))
  Using cached https://files.pythonhosted.org/packages/58/f8/e2cca22387965584a409795913b774235752be4176d276714e15e1a58884/starlette-0.27.0-py3-none-any.whl
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from attrs>=17.3.0->aiohttp->-r requirements.txt (line 3)) (6.7.0)
Requirement already satisfied: idna>=2.0 in /usr/lib/python3/dist-packages (from yarl<2.0,>=1.0->aiohttp->-r requirements.txt (line 3)) (2.6)
Collecting exceptiongroup; python_version < "3.11" (from anyio<4.0.0,>=3.7.1->fastapi->-r requirements.txt (line 9))
  Downloading https://files.pythonhosted.org/packages/ad/83/b71e58666f156a39fb29417e4c8ca4bc7400c0dd4ed9e8842ab54dc8c344/exceptiongroup-1.1.3-py3-none-any.whl
Collecting sniffio>=1.1 (from anyio<4.0.0,>=3.7.1->fastapi->-r requirements.txt (line 9))
  Downloading https://files.pythonhosted.org/packages/c3/a0/5dba8ed157b0136607c7f2151db695885606968d1fae123dc3391e0cfdbf/sniffio-1.3.0-py3-none-any.whl
Collecting pydantic-core==2.6.3 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi->-r requirements.txt (line 9))
  Using cached https://files.pythonhosted.org/packages/cb/fe/8c9363389f8f303fb151895af83ac30e06c0406779fe188b4281a64e4c50/pydantic_core-2.6.3.tar.gz
  Installing build dependencies ... done
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib/python3.7/tokenize.py", line 447, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-r73yhlap/pydantic-core/setup.py'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-r73yhlap/pydantic-core/

Even though the installation of PrintWatchAI_Backend was not successful, I installed the plugin on DWC, but once I enter the data in "settings" when I regenerate the page, they are not saved and you have to enter them again.
In the "preview" screen the webcam does not work.
DWC AI.jpg

For now I don't have any other tests I can do...