Introduction
Hello all, we are looking to gauge interest and get feedback on an AI integration for RRF.

Features

• Visual Failure monitoring: uses frames from the webcam/IP camera/RTSP camera and detects spaghetti failure and notifies the users OR pauses/takes action on behalf of the printer.

• Anomaly Detection: using ObjectModel data from the printer, monitors and tracks the current health of the printer and it's subcomponents. Used for scheduling preventative maintenance and detecting+diagnoising issues with the printer.

Comments
If you are interested in a feature/plugin/integration that provides you access with these AI features, please like or comment on this thread.

If you have requests for features that aren't listed and fall under the scope of AI + Data, please leave a comment or reach out via direct message/email.

Contact
Email: Lebiedzinskip@printpal.io

To all who upvoted (and those who didn't but are still interested): we have the working beta version of the AI monitoring plugin available for testing. Currently, the plugin will only work on Duet printers running in SBC mode.

The ZIP install file for the plugin may be found on GitHub: https://github.com/printpal-io/DSF_PrintWatchAI_Plugin

Please reach out to me personally (lebiedzinskip@printpal.io) to receive a free API key for testing without limits. The free version without the API key will only allow you up to 12 hours of AI monitoring.

This is what the plugin currently looks like, any feedback is taken seriously, and very much appreciated:

@T3P3Tony

Of course, thank you for reminding me.

The minimum requirements are as follows:

Visual Failure Monitoring:

• Firmware Requirements:

  • RepRapFirmware 3.4.5b3+ and above, this firmware contains support for the application sending commands to the printer when a failure has occurred. The software will still work and be able to notify you of failures if not using this firmware, however no actions will be able to be taken on behalf of the printer.

• Supported formats:

  • HTTP/S or MJPEG stream (preferred)
  • RTSP
  • USB
  • PiCamera

• Resolution:

  • 480 x 480 pixels @ 100% quality (no compression) --> 768 x 768 @ 100% quality or greater recommended

• Hardware:

  • Able to process 1 image every 10 seconds
  • Internet connected (Ethernet or WiFi)
  • Can communicate with the Duet3D board
  • Examples:
    • Raspberry Pi 3
    • Raspberry Pi 4
    • Raspberry Pi zero W
    • ESP32 - does not have ethernet option to communicate with Duet3D board
  • Edge option:
    • Embedded AI is an option and the hardware will fit inside the electronics cabinet of the printer. For this you'd have to contact us.

Anomaly Detection:

• Firmware requirements:
  • Most recent firmware with support for datalogging (TBD)
• Hardware:
  • Although this should be available for everyone in some capacity, hardware that provides more feedback information is better, this includes, but is not limited to:
    • Closed-loop steppers/servos
    • Accelerometer
    • Lidar
    • Temperature sensors
    • Flow sensors
    • Voltage/Amperage sensors

@CNCModeller I cannot say too much, but we have been working with 3D Convolutions and Pointclouds of various CAD models, and there are very exciting things coming in the future.

@DNvM84 @jay_s_uk
I've created a Plugin for DWC-only configurations to be able to run the AI monitoring. It will work for DWC 3.4, 3.4.6, and 3.5.0-rc as of now. It can be found here: https://github.com/printpal-io/DWC_PrintWatchAI_Plugin

In order to use the plugin, you still need a device (Raspberry Pi, or any SBC) running the AI backend component, and it needs to be on the same network (Ethernet or WiFi) as the Duet mainboard. The backend component as well as how to set it up can be found here: https://github.com/printpal-io/PrintWatchAI_Backend

I would recommend using a Raspberry Pi for running the backend since they are simple and easy to use.
I have also tested the Plugin's Backend component with great success on an Orange Pi Zero3 board and it works just as well. The Orange Pi Zero3 is a little more economical ($22.99 on amazon: https://www.amazon.com/Orange-Pi-Allwinner-Quad-Core-Development/dp/B0CB1BYTT8/ref=sr_1_5?crid=BYJ6JUDXT64O&keywords=orange+pi+zero+3&qid=1694217352&sprefix=orange+pi+zero+3%2Caps%2C118&sr=8-5)

When testing, please run with Test Mode enabled, since the code that checks if the printer is 'Printing' has not been fully tested.

@chrishamm This new firmware has fixed the issue. Thank you.

@Gianluca
We have created a ZIP installer for the 3.5.0-rc.1 versions of RRF/DSF/DWC, it can be found here: https://github.com/printpal-io/DSF_PrintWatchAI_Plugin/releases/tag/v3.5.0-rc.1

Please note that the 3.5.0-rc.1 currently has an issue with the HTTP endpoints being registered. This can be easily worked around by doing the following:

• Make sure the plugin is installed and started
• SSH into the SBC
• Enter the command: sudo systemctl restart duetwebserver

This will restart the webserver and register the endpoints correctly. This must be done every time the SBC is started up. This should be fixed in the next release.

@DNvM84 I would recommend placing the camera off of any corner of the print bed, with the center of the frame pointed at where the majority of the print will take.

I would also recommend placing it above the print bed such that it is 15 - 45 degrees from the vector of the plane that defines the print bed:

The angle can be determined by what you are printing and by how it works with the toolhead on your machine. Play around and find out what works best. Also, if printing an item that has a highly detailed infill that may confuse the AI, you would want to have the camera at a lower/higher angle so it does not perceive the infill as highly dense details.

@Gianluca I will create a ZIP installer for your version of DSF. I will get back to you within the hour.

@deckingman This would depend on what sensor values are available. If we have feedback data from the extruder motor (current, position, command position, etc), then yes. That is why we like the way Duet is moving with adding support for closed-loop steppers and low-level variable sampling at 4 Hz.

As for diagnosing the root cause of the issue, maybe, but not with high certainty, at least not as of now with our testing. We plan on trying various simple methods first before developing an end-to-end model for this, some of which would be clustering the data from our AD ("Anomaly Detection") model and seeing if we can segment out any distinct clusters that we can label as "pertaining to an issue with X, Y, or Z".

Regarding your last question, if it is a computer vision model then it is very unlikely to be able to discriminate genuine defects vs model features. It takes everything at face value as it sees it on the sensor. This of course could be addressed with a CAD model informed ML model, but we do not have plans on developing that as of now.

@Herve_Smith This Forum post was made prior to us planning on supporting DWC-only, since then we have developed a plugin that works with DWC-only configurations, see forum comment:

@Peter-Lebiedzinski said in AI Print Monitoring / Printer Health monitoring:

@DNvM84 @jay_s_uk
I've created a Plugin for DWC-only configurations to be able to run the AI monitoring. It will work for DWC 3.4, 3.4.6, and 3.5.0-rc as of now. It can be found here: https://github.com/printpal-io/DWC_PrintWatchAI_Plugin

In order to use the plugin, you still need a device (Raspberry Pi, or any SBC) running the AI backend component, and it needs to be on the same network (Ethernet or WiFi) as the Duet mainboard. The backend component as well as how to set it up can be found here: https://github.com/printpal-io/PrintWatchAI_Backend

I would recommend using a Raspberry Pi for running the backend since they are simple and easy to use.
I have also tested the Plugin's Backend component with great success on an Orange Pi Zero3 board and it works just as well. The Orange Pi Zero3 is a little more economical ($22.99 on amazon: https://www.amazon.com/Orange-Pi-Allwinner-Quad-Core-Development/dp/B0CB1BYTT8/ref=sr_1_5?crid=BYJ6JUDXT64O&keywords=orange+pi+zero+3&qid=1694217352&sprefix=orange+pi+zero+3%2Caps%2C118&sr=8-5)

When testing, please run with Test Mode enabled, since the code that checks if the printer is 'Printing' has not been fully tested.

@Gianluca said in AI Print Monitoring / Printer Health monitoring:

You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.

You may need to upgrade pip using python.

See if this post helps at all: https://github.com/pypa/pip/issues/5599

or if any of these commands work:
python3 -m pip install --upgrade pip
python -m pip install --upgrade pip

pfmoore created this issue in pypa/pip

closed ImportError in system pip wrappers after an upgrade #5599

@Gianluca This looks like it might be an issue with the version of pip installed. Can you try updating it with:
pip3 install --upgrade pip
or
pip install --upgrade pip

and then re-running the installation commands that failed for the backend?

None of the DWC plugin settings or UI items will work if the Backend is not running so both of your issues should be fixed by doing this. Let me know if this fixes your issue.

@DNvM84 @jay_s_uk
I've created a Plugin for DWC-only configurations to be able to run the AI monitoring. It will work for DWC 3.4, 3.4.6, and 3.5.0-rc as of now. It can be found here: https://github.com/printpal-io/DWC_PrintWatchAI_Plugin

In order to use the plugin, you still need a device (Raspberry Pi, or any SBC) running the AI backend component, and it needs to be on the same network (Ethernet or WiFi) as the Duet mainboard. The backend component as well as how to set it up can be found here: https://github.com/printpal-io/PrintWatchAI_Backend

I would recommend using a Raspberry Pi for running the backend since they are simple and easy to use.
I have also tested the Plugin's Backend component with great success on an Orange Pi Zero3 board and it works just as well. The Orange Pi Zero3 is a little more economical ($22.99 on amazon: https://www.amazon.com/Orange-Pi-Allwinner-Quad-Core-Development/dp/B0CB1BYTT8/ref=sr_1_5?crid=BYJ6JUDXT64O&keywords=orange+pi+zero+3&qid=1694217352&sprefix=orange+pi+zero+3%2Caps%2C118&sr=8-5)

When testing, please run with Test Mode enabled, since the code that checks if the printer is 'Printing' has not been fully tested.

@Gianluca
We have created a ZIP installer for the 3.5.0-rc.1 versions of RRF/DSF/DWC, it can be found here: https://github.com/printpal-io/DSF_PrintWatchAI_Plugin/releases/tag/v3.5.0-rc.1

Please note that the 3.5.0-rc.1 currently has an issue with the HTTP endpoints being registered. This can be easily worked around by doing the following:

• Make sure the plugin is installed and started
• SSH into the SBC
• Enter the command: sudo systemctl restart duetwebserver

This will restart the webserver and register the endpoints correctly. This must be done every time the SBC is started up. This should be fixed in the next release.

@CNCModeller This would be helpful, but not necessary. At the end of the day we want to make it as modular and easy to use for the user.

With nowadays computer vision operations, we can easily pointcloud the current print from a single camera angle.

We could perhaps wrap this into a feature rather quickly and release it for beta testing. The end feature would be a detector for clog, something conventional object detectors struggle to detect as it is trying to detect the absence of something in the frame.

@deckingman This would depend on what sensor values are available. If we have feedback data from the extruder motor (current, position, command position, etc), then yes. That is why we like the way Duet is moving with adding support for closed-loop steppers and low-level variable sampling at 4 Hz.

As for diagnosing the root cause of the issue, maybe, but not with high certainty, at least not as of now with our testing. We plan on trying various simple methods first before developing an end-to-end model for this, some of which would be clustering the data from our AD ("Anomaly Detection") model and seeing if we can segment out any distinct clusters that we can label as "pertaining to an issue with X, Y, or Z".

Regarding your last question, if it is a computer vision model then it is very unlikely to be able to discriminate genuine defects vs model features. It takes everything at face value as it sees it on the sensor. This of course could be addressed with a CAD model informed ML model, but we do not have plans on developing that as of now.

@CNCModeller I cannot say too much, but we have been working with 3D Convolutions and Pointclouds of various CAD models, and there are very exciting things coming in the future.

@Gianluca I will create a ZIP installer for your version of DSF. I will get back to you within the hour.

@DNvM84 I would recommend placing the camera off of any corner of the print bed, with the center of the frame pointed at where the majority of the print will take.

I would also recommend placing it above the print bed such that it is 15 - 45 degrees from the vector of the plane that defines the print bed:

The angle can be determined by what you are printing and by how it works with the toolhead on your machine. Play around and find out what works best. Also, if printing an item that has a highly detailed infill that may confuse the AI, you would want to have the camera at a lower/higher angle so it does not perceive the infill as highly dense details.