Ok I think I found the issue. I had initially powered the Raspberry Pi from the duet 3. Then I read its recommended to use an extra PSU in some cases so I then powered them separately. However I did not adjust the jumpers on the board to accommodate for this. I have now adjusted it so that only the INT 5v EN jumper is wired in.
Now I have a constant temperature read out of the MCU and the SBC connects to the board every-time.