Duet 3 Mini 5+ VIN under-voltage issues in SBC mode

PaulHew

@droftarts Ian, SBC mode.

Fletcher

I'm seeing this too. DuetMini5+ Ethernet in SBC mode (RPi 3B+), M115 for mainboard and toolboard below:

M115
FIRMWARE_NAME: RepRapFirmware for Duet 3 Mini 5+ FIRMWARE_VERSION: 3.3beta1 ELECTRONICS: Duet 3 Mini5plus Ethernet FIRMWARE_DATE: 2021-02-14 16:24:45
M115 B121
Duet TOOL1LC firmware version 3.3beta1 (2021-02-14 16:34:04)

I left my printer sit powered on this week, and see 3 VIN undervolt events while idle. It also died again 3 hours into a 5 hour print last night. I'm happy to do whatever troubleshooting would help folks gather more info to get the root cause nailed down.

dc42

@fletcher said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

I left my printer sit powered on this week, and see 3 VIN undervolt events while idle.

That's the first time we have seen this issue when the printer is idle. Perhaps it just wasn't noticed before. What was the voltage reported with the undervoltage events: 0.0, or a higher value?

Fletcher

@dc42

Also 0.0v. It's possible I mis-read my logs, but I am pretty sure of what I saw. Also of note: for at least some of that time, my heater was on because I am an idiot (but thankfully I didn't burn my house down). Of course I closed the RRF tab in Firefox this morning because my laptop was dragging -- is there any way to retrieve historical console data from SBC, so I could potentially see exactly when those messages happened and if the heater was on at the time?

[edit]On further thought, I'd say it's more likely that I misread my log, than that this is a new problem you haven't seen before. I'm going to leave my machine on and idle again with a console window up and see if I can capture this again.[/edit]

[edit]I can, in fact, Google, so I've enabled logging with M929 S1 P"vin-debug.log" and we'll see what we get. Is there any harm in adding this to my config.g considering I'm running on a 32gb SD card and there's no way I'm anywhere close to filling that? Would be nice if we could do auto log rollover (ie: save to a new filename after the file reaches a certain size) but c'est la vie! [/edit]

Serbitar

@fletcher You can probably get logs from the SBC:

sudo journalctl  --unit duetcontrolserver

Fletcher

Ok, I caught a VIN error while idle! See below: The error at 1:42 AM was the print failing. The error at 4:43 PM was the machine sitting idle.

PaulHew

Happened again Mid-Print.

Had to issue a m564 h0 so I could lower the bed and clear the bed.

Again, SBC mode, rrf3.2.2

P.

droftarts

@fletcher @paulhew We’ve been testing various new firmwares from @dc42 over the last week, but there’s still an issue. The trouble with this bug is that it can take 24hrs+ to happen! As far as we are aware, it does not happen in Standalone mode, so rather than waste filament on failed prints and raise the level of frustration, disconnect the SBC.

Ian

Fletcher

@droftarts Overnight, I had two more events:

I've tried two different PSUs and replaced 100% of the 24v wiring (before finding this thread), so I'm fairly certain it's not a power supply or wiring issue. Only remaining thing I could swap out is the 120V from the power entry module to the PSU, but this really feels like the same issue.

Good news is, I've got a build that seems to reliably generate failures every 12 hours, if you need more beta testers for firmware!

Due to the way my space is laid out, I'd need a wifi bridge (or to re-configure my SBC as a wifi bridge, I suppose) to use the Mini5 Ethernet without the SBC, so I'm down to test firmware for RRF.

ukwebb

I too ( as previously mentioned ) was having this issue , and to test the undervolt side I've swapped back to standalone mode, but kept everything else the same, so the pi4 is still running,its still running Duet interface ( although it cant connect! ), its still running the 7" touch hdmi screen and C270 webcam etc etc etc. The only difference is the onboard SD card, and disconnected the GPIO cable.
2 weeks have now passed and not a single undervolt issue.

dc42

@ukwebb said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

I too ( as previously mentioned ) was having this issue , and to test the undervolt side I've swapped back to standalone mode, but kept everything else the same, so the pi4 is still running,its still running Duet interface ( although it cant connect! ), its still running the 7" touch hdmi screen and C270 webcam etc etc etc. The only difference is the onboard SD card, and disconnected the GPIO cable.
2 weeks have now passed and not a single undervolt issue.

Thanks, this confirms what we suspected, that it only happens in SBC mode.

dc42

I have identified and (I believe) fixed an issue that was contributing to this problem. For anyone who would like to try it, there is a new firmware build at https://www.dropbox.com/sh/b65da04w1gl94nf/AACLpPslfDiWDlJN5KDdPydAa?dl=0.

PaulHew

@dc42 Thanks as always for your hard work.
Have loaded and will let you know..

M-Pixel

+1 over here. Duet 3 Mini 5+ in SBC mode; SBC is powered by a step-down which is in turn powered by the same 48V supply that powers the Duet, so it's not possible for the Duet to lose power without the SBC also losing power. Occurrence seems completely random - I haven't observed a single common variable between the occurrences (apart from the fact that they all occur on the same printer, of course).

I've been contending with the issue by figuring out approximately where the print stopped, then manually editing the gcode file to pick up where it left off.

I'm loading the April 7th firmware now, and will post diagnostics if the issue occurs again.

Fletcher

@dc42 Thank you! I loaded your 7APR firmware at 10:33 AM local time today and left the board sitting idle while I worked. It's now 18:50 with zero VIN under-voltage events, whereas I had at least one during this time-frame each of the last two days (with the machine sitting idle). I'm going to let the machine sit for another 24 hours before I try printing, but, signs are positive!

PaulHew

Morning. After loading the firmware last night, I was greeted by this this morning. Machine was idle.

E-Stopped the board and has stopped the VSSA faults.

Have some tuning to do this morning, so will report back.

Regards,
Paul.

dc42

@paulhew, please run M115 to confirm the firmware build date/time.

PaulHew

@dc42

m115
FIRMWARE_NAME: RepRapFirmware for Duet 3 Mini 5+ FIRMWARE_VERSION: 3.3beta2+1 ELECTRONICS: Duet 3 Mini5plus Ethernet FIRMWARE_DATE: 2021-04-07 13:33:17

Regards,

Paul

PaulHew

@paulhew Just had these pop up

They are new!

edit: and it did all of the movements and heating but nothing on the bed.
No blockage. Going back to 3.2.2

edit.edit: was I supposed to upgrade the 1LC firmware also?

Downloaded 3.2.2, upload to board, now I get this.

M997 S0
Error: M997: Failed to find IAP file /opt/dsf/sd/firmware/Duet3_SBCiap32_Mini5plus.bin

P.

dc42

@paulhew said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

edit.edit: was I supposed to upgrade the 1LC firmware also?

Yes, if you run 3.3 on the main board then you need 3.3 on expansion and tool boards too.