Dynamic G-Code Modification Based on Sensor Feedback

[username19]

Repost due to the first post being locked:
Hi all,

I'm working on a project to create an automated gantry system for palletizing bags in a warehouse. The system uses a CNC kit controlled by a Duet 2 board, a Raspberry Pi for processing, an ultrasonic sensor on the end effector, and a camera to monitor pallet zones. The end effector is a vacuum-based mechanism for picking up bags.

One challenge lies in dynamically altering the G-code based on real-time sensor feedback to automate the gantry's movements. Here's what I'm aiming to achieve:

Process camera images with the Raspberry Pi to determine the location of bags and pallets.
Use ultrasonic sensor data to adjust the Z-axis movement accurately.
Modify the G-code in real-time based on these inputs for precise positioning and movement of the gantry.
I would greatly appreciate any advice, resources, or examples of how to dynamically update G-code based on sensor feedback. Specifically, how to interface the Raspberry Pi with the Duet 2 controller for this purpose and any software or scripting recommendations to achieve this level of automation.

Looking forward to your insights and suggestions.

Thanks in advance!

[T3P3Tony]

@Rackan-Mansour the original thread is unlocked so best to discuss there.