Duet 3 Mini 5+ VIN under-voltage issues in SBC mode

dc42

@carcamerarig, thanks for the additional data point.

My best guess at this point is that the DMA for the SPI transfers between the Pi and the Duet is somehow interfering with the DMA used by the ADC. Do any of you experiencing this problem see short-lived temperature jumps too?

Serbitar

@dc42 by the time i'm back at the printer, usually several hours have past since it stopped so the temperature charts have gone past.

I have a few prints to complete in standalone mode and then i will enable the extra monitoring. Is there any way to save logs of the temp values too?

dc42

Until I have pinned down the exact cause, it might be worth trying a lower SPI frequency. The frequency is set in the DCS configuration file on the Pi.

dc42

PS - if anyone who has experienced this has their machine idle, please can you run some simulated prints, to see whether they can trigger this issue. The increased SPI traffic rate from the Pi during a simulation might make it happen more often. OTOH it could turn out that step interrupts also need to happen for this issue to occur.

WILD9

@dc42 Happy to try, how would this manifest during a simulation, just console messages showing low voltage detected? In which case I will make sure I have 24Vin active during simulation.

dc42

@WILD9 said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

@dc42 Happy to try, how would this manifest during a simulation, just console messages showing low voltage detected? In which case I will make sure I have 24Vin active during simulation.

I expect the indications of the problem to be:

A console report of an under-voltage event, with the reported VIN being 0.0
If you run M122 afterwards then it will report that one or more under-voltage events occurred and the minimum recorded VIN will be 0.0. Note, a subsequent M122 will only report under-voltage events and the minimum VIN that happened since the previous M122.

You will need to have VIN power applied throughout.

carcamerarig

@dc42 said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

PS - if anyone who has experienced this has their machine idle, please can you run some simulated prints, to see whether they can trigger this issue. The increased SPI traffic rate from the Pi during a simulation might make it happen more often. OTOH it could turn out that step interrupts also need to happen for this issue to occur.

I’ve been spending quite a few hours daily of stop start printing while I’m still tuning and configuring (mostly learning) and have only seen the one which was at a home all, the effect was a crash into the left x carriage carrier, I’m unsure if the error was before during or after the crash.
I’ll be doing more of the same later if you want me to test or log anything, I’ll need some instructions though if I’m to alter some code.

Serbitar

@dc42 Is 4MHz low enough?

lodger

Is it worth running some heaters duing a simulation. To give the PSU some load.
During a print my VIN can jump up and down a few points of a volt. But during a sim . She is bang on 24.2v and dont move.

WILD9

6ish hours of simulation so far with no issues

droftarts

@WILD9 I ran simulations for 8 hours last night, with no issues. Still trying to track this issue down.

Ian

lodger

I have come down this morning to find a stalled print. Here is my M122,

m122

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.2.2 running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: XJDX9-1296U-D65J0-40KMJ-4K03Z-7J6X6
Used output buffers: 1 of 40 (18 max)
=== RTOS ===
Static ram: 98732
Dynamic ram: 95844 of which 64 recycled
Never used RAM 50576, free system stack 122 words
Tasks: Linux(ready,99) HEAT(blocked,297) CanReceiv(blocked,947) CanSender(blocked,358) CanClock(blocked,362) TMC(blocked,75) MAIN(running,476) IDLE(ready,20) AIN(blocked,264)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 17:19:34 ago, cause: software
Last software reset at 2021-03-19 15:58, reason: User, none spinning, available RAM 50808, slot 1
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0400f000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
Supply voltage: min 0.0, current 24.2, max 24.3, under voltage events: 2, over voltage events: 0, power good: yes
Driver 0: position 16355, standstill, SG min/max 0/428, read errors 0, write errors 1, ifcnt 223, reads 13005, writes 43, timeouts 17110, DMA errors 0, failedOp 0x71
Driver 1: position 2458, standstill, SG min/max 0/410, read errors 0, write errors 1, ifcnt 223, reads 30107, writes 43, timeouts 7, DMA errors 0, failedOp 0x6f
Driver 2: position 63048, standstill, SG min/max 0/44, read errors 0, write errors 1, ifcnt 189, reads 29631, writes 32, timeouts 494, DMA errors 0, failedOp 0x71
Driver 3: position 0, standstill, SG min/max 0/30, read errors 0, write errors 1, ifcnt 188, reads 29889, writes 32, timeouts 235, DMA errors 0, failedOp 0x01
Driver 4: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 155, reads 30130, writes 27, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Date/time: 2021-03-20 09:18:05
Cache data hit count 4294967295
Slowest loop: 178.10ms; fastest: 0.08ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 11368610ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 190766, completed moves 190766, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = 2 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 561364, send timeouts 561361, received 0, lost 0, longest wait 0ms for reply type 0, free buffers 15
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 3ms ago
RX/TX seq numbers: 42328/42328
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x10eec
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.2.2
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 36.40
Maximum length of RX/TX data transfers: 5108/1664.

dc42

@lodger, was that a print or a simulation?

Please can you try the firmware at https://www.dropbox.com/sh/6203euyyobllic5/AABTHIbn7F_maK9p-oy7vq3Ea?dl=0 with the same file to see if it happens again.

lodger

@dc42 Hi,

Yes it was a print. I will just put the kettle on and update and print. It was a 10 hour print so i may not get back to you till tomo.

Edit:: I have updated and run a Similator of the same file. Without problems !
Just running the print now .

lodger

@dc42 Mornings.

The print completed . with no under voltage triggers at all. I will try print something else and report if i get any more problems

dc42

@lodger said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

@dc42 Mornings.

The print completed . with no under voltage triggers at all. I will try print something else and report if i get any more problems

Thanks for the update.

Anyone else having this problem and happy to use beta firmware, feel free to try the firmware that I linked to above. It's compatible with DWC 3.3beta2.

Serbitar

I have put my machine back into SBC mode running the linked beta firmware.

It completed a 12h print over night, print queue has shorter jobs for the next few days. I will update if there are any under voltage detection issues.

dc42

@Serbitar, thanks for the update.

Serbitar

Not fixed i'm afraid

Mar 24 19:21:32 DuetControlServer[3177]: Print auto-paused due to low voltage
Mar 24 19:21:32 DuetControlServer[3177]: [warn] VIN under-voltage event (0.0V)

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.3beta2+1 running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: W4ZW4-V296U-D65J0-40KM0-LR03Z-H6928
Used output buffers: 1 of 40 (32 max)
=== RTOS ===
Static ram: 99720
Dynamic ram: 98408 of which 24 recycled
Never used RAM 45552, free system stack 88 words
Tasks: Linux(ready,145) HEAT(notifyWait,299) CanReceiv(notifyWait,943) CanSender(notifyWait,358) CanClock(delaying,340) TMC(notifyWait,109) MAIN(running,310) IDLE(ready,20) AIN(notifyWait,260)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 21:38:59 ago, cause: software
Last software reset at 2021-03-23 21:38, reason: User, none spinning, available RAM 46528, slot 2
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
Supply voltage: min 24.1, current 24.2, max 24.4, under voltage events: 3, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Driver 0: position -5573, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 19, reads 5889, writes 9, timeouts 17, DMA errors 0, failedOp 0x01
Driver 1: position -6035, standstill, SG min/max 0/44, read errors 0, write errors 0, ifcnt 20, reads 5906, writes 9, timeouts 0, DMA errors 0
Driver 2: position 8601, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 19, reads 5906, writes 9, timeouts 0, DMA errors 0
Driver 3: position 0, standstill, SG min/max 0/256, read errors 0, write errors 0, ifcnt 22, reads 5906, writes 9, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, SG min/max 0/210, read errors 0, write errors 0, ifcnt 22, reads 5906, writes 9, timeouts 0, DMA errors 0
Driver 5: position 0, standstill, SG min/max 0/2, read errors 0, write errors 0, ifcnt 22, reads 5906, writes 9, timeouts 0, DMA errors 0
Driver 6: position 0, standstill, SG min/max 2/2, read errors 0, write errors 0, ifcnt 22, reads 5906, writes 9, timeouts 0, DMA errors 0
Date/time: 2021-03-24 19:22:01
Cache data hit count 4294967295
Slowest loop: 1.95ms; fastest: 0.13ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 467805, completed moves 467805, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 1], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.5
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is doing "G1 X-68.586998 Y-74.568001 E0.009290" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Filament sensors ===
Extruder 0 sensor: ok
Extruder 1 sensor: ok
=== CAN ===
Messages queued 266, send timeouts 266, received 0, lost 0, longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 17 (min 17)
Last cancelled message type 30 dest 127
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 5ms ago
RX/TX seq numbers: 65096/65096
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x11578
Buffer RX/TX: 1776/3288-0
=== Duet Control Server ===
Duet Control Server v3.3-b2
File:
Buffered code: G1 X-68.587 Y-74.568 E0.00929
Buffered code: G1 X-67.933 Y-73.726 E0.01535
Buffered code: G1 X-67.168 Y-72.475 E0.02112
Buffered code: G1 X-66.365 Y-70.761 E0.02725
Buffered code: G1 X-65.747 Y-68.922 E0.02792
Buffered code: G1 X-67.640 Y-69.345 E0.02792
Buffered code: G1 X-69.429 Y-69.963 E0.02725
Buffered code: G1 X-70.754 Y-70.594 E0.02112
Buffered code: G1 X-71.659 Y-71.156 E0.01535
Buffered code: G1 X-72.167 Y-71.554 E0.00929
Buffered code: G1 X-72.601 Y-72.094 E0.00997
Buffered code: G1 X-75.105 Y-70.343 E0.04398
Buffered code: G1 X-74.694 Y-69.718 E0.01078
Buffered code: G1 X-74.473 Y-69.112 E0.00928
Buffered code: G1 X-74.219 Y-68.077 E0.01535
Buffered code: G1 X-74.028 Y-66.623 E0.02111
Buffered code: G1 X-73.992 Y-64.730 E0.02725
Buffered code: G1 X-74.175 Y-62.799 E0.02792
Buffered code: G1 X-75.733 Y-63.955 E0.02792
Buffered code: G1 X-77.116 Y-65.248 E0.02725
Buffered code: G1 X-78.069 Y-66.363 E0.02112
Buffered code: G1 X-78.668 Y-67.244 E0.01535
Buffered code: G1 X-78.970 Y-67.814 E0.00929
Buffered code: G1 X-79.146 Y-68.484 E0.00997
Buffered code: G1 X-82.146 Y-67.903 E0.04398
Buffered code: G1 X-82.025 Y-67.165 E0.01077
Buffered code: G1 X-82.070 Y-66.521 E0.00929
Buffered code: G1 X-82.258 Y-65.472 E0.01535
Buffered code: G1 X-82.675 Y-64.066 E0.02112
Buffered code: G1 X-83.413 Y-62.322 E0.02725
Buffered code: G1 X-84.365 Y-60.633 E0.02792
Buffered code: G1 X-85.318 Y-62.322 E0.02792
==> 1512 bytes
Code buffer space: 1660
Configured SPI speed: 8000000 Hz
Full transfers per second: 47.08
Codes per second: 17.66
Maximum length of RX/TX data transfers: 5884/1664
File /opt/dsf/sd/gcodes/eggGears-2and6_0.08mm_PETG_ENDER3_6h50m.gcode is selected, processing

ukwebb

Just to add another voice, I and also getting this same issue.

Last night, 70% and 5 hours into a print it just stopped, and displayed VIN undervoltage event (0.0v) and then Insufficient axis homed

My setup is based around a coreXY machine, with a dedicated meanwell 5v 5a psu for the pi, and a meanwell 24v 6amp supply for the printer, the hot bed is 240v

Also, this hasn't happed until recently - i think after i did an apt update. so it was probably fine on 3.1 but not 3.2