Duet 3 Mini 5+ VIN under-voltage issues in SBC mode

[PaulHew]

@dc42 Just out of interest, what is the critical bit in our logs that assist you DC42?

[PaulHew]

Undervotage.... Printer was idle.

13/04/2021, 09:18:15 	m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.3beta2+1 (2021-04-12 08:36:30) running on Duet 3 Mini5plus Ethernet (SBC mode)
Board ID: A45XG-F396U-D65J0-40KML-1F03Z-H03XP
Used output buffers: 1 of 40 (23 max)
=== RTOS ===
Static ram: 100072
Dynamic ram: 91932 of which 0 recycled
Never used RAM 51700, free system stack 172 words
Tasks: Linux(resourceWait,119) HEAT(delaying,322) CanReceiv(notifyWait,774) CanSender(notifyWait,371) CanClock(delaying,340) TMC(notifyWait,99) MAIN(running,484) IDLE(ready,20) AIN(notifyWait,259)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 12:02:11 ago, cause: software
Last software reset at 2021-04-12 21:16, reason: User, none spinning, available RAM 51700, slot 2
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU revision 3, ADC conversions started 51519632, completed 51519631, timed out 0, errs 3
tnd=0 rnd=0 wrd=0 wtd=3 war=3
Supply voltage: min 0.0, current 23.9, max 24.0, under voltage events: 1, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Driver 0: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 82, reads 28663, writes 20, timeouts 51838, DMA errors 0, failedOp 0x6f
Driver 1: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 82, reads 14925, writes 20, timeouts 40, DMA errors 0, failedOp 0x06
Driver 2: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 84, reads 14965, writes 20, timeouts 0, DMA errors 0
Driver 3: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 84, reads 14963, writes 20, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 84, reads 14965, writes 20, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Date/time: 2021-04-13 09:18:16
Cache data hit count 4294967295
Slowest loop: 63.01ms; fastest: 0.10ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 389993, send timeouts 0, received 519973, lost 0, longest wait 2ms for reply type 6049, peak Tx sync delay 500, free buffers 17 (min 16)
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 4ms ago
RX/TX seq numbers: 36429/36429
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x118a8
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.2.2
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 35.63
Maximum length of RX/TX data transfers: 3284/604
12/04/2021, 23:04:48 	Warning: VIN under-voltage event (0.0V)

[Fletcher]

@dc42 Another VSSA fault chain on the latest firmware:

4/13/2021, 7:41:44 AM 	Error: VSSA fault, check thermistor wiring
4/13/2021, 7:41:44 AM 	m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.3beta2+1 (2021-04-12 08:36:30) running on Duet 3 Mini5plus Ethernet (SBC mode)
Board ID: FDQG2-Q296U-D65J0-40KMG-1K03Z-HXFTZ
Used output buffers: 1 of 40 (12 max)
=== RTOS ===
Static ram: 100072
Dynamic ram: 91764 of which 64 recycled
Never used RAM 51804, free system stack 172 words
Tasks: Linux(ready,137) HEAT(delaying,322) CanReceiv(notifyWait,774) CanSender(notifyWait,371) CanClock(delaying,340) TMC(notifyWait,99) MAIN(running,498) IDLE(ready,20) AIN(notifyWait,260)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 23:27:06 ago, cause: software
Last software reset at 2021-04-12 16:13, reason: User, none spinning, available RAM 51856, slot 1
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU revision 3, ADC conversions started 100481305, completed 100481304, timed out 0, errs 1
tnd=0 rnd=0 wrd=0 wtd=1 war=1
Supply voltage: min 0.0, current 0.1, max 24.4, under voltage events: 1, over voltage events: 0, power good: no
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Driver 0: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 218, reads 4487, writes 11, timeouts 0, DMA errors 0
Driver 1: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 220, reads 4487, writes 11, timeouts 0, DMA errors 0
Driver 2: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 192, reads 4489, writes 9, timeouts 0, DMA errors 0
Driver 3: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 218, reads 4486, writes 11, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 218, reads 4487, writes 11, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Date/time: 2021-04-13 15:41:41
Cache data hit count 4294967295
Slowest loop: 57.18ms; fastest: 0.08ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 759838, send timeouts 0, received 1013078, lost 0, longest wait 2ms for reply type 6049, peak Tx sync delay 354, free buffers 17 (min 16)
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 4ms ago
RX/TX seq numbers: 14877/14877
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x118a8
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.3-b2
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 35.89
Codes per second: 0.00
Maximum length of RX/TX data transfers: 3044/804
4/13/2021, 7:41:39 AM 	Error: VSSA fault, check thermistor wiring

It was stable for almost 24 hours, which is positive change, at least! Thank you for all the work you're putting in on this issue.

[dc42]

All, please try the new firmware at https://www.dropbox.com/sh/0mwfe22wepk5efh/AAB6Lp0rwBPjVK2R4SxbKhLya?dl=0. M115 reports it as 3.2beta2+2.

This version avoids using the ADC DMA sequencer completely, so I am hopeful that it will solve the problem. I was worried that the additional ADC interrupt load would be excessive, but with careful code optimisation I have got it down to 0.5% of total CPU time.

[Serbitar]

@dc42 Looks like this version (and probably the other recent ones) breaks duetcontrolserver.

With 3.3b2 the logs are nice and quiet, with 3.3b2+2 the duetcontrolserver log is a fast rolling spam like this:

Apr 13 20:58:38 ender3 DuetControlServer[1585]: [error] Failed to merge JSON due to internal error: {"key":"","flags":"d99fn","result":{"boards":[{"mcuTemp":{},"vIn":{"current":0.4}}],"fans":[],"heat":{"heaters":[]},"inputs":[{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"}],"job":{"build":null,"duration":null,"filePosition":0,"layer":null,"layerTime":null,"pauseDuration":null,"timesLeft":{"filament":null,"file":null,"slicer":null},"warmUpDuration":null},"move":{"axes":[{"machinePosition":0,"userPosition":0},{"machinePosition":0,"userPosition":0},{"machinePosition":0,"userPosition":0}],"currentMove":{"acceleration":0,"deceleration":0,"laserPwm":null,"requestedSpeed":0,"topSpeed":0},"extruders":[],"virtualEPos":0},"sensors":{"analog":[],"endstops":[null,null,null],"filamentMonitors":[],"gpIn":[],"probes":[]},"seqs":{"boards":0,"directories":0,"fans":0,"global":0,"heat":0,"inputs":77,"job":1,"move":2,"network":2,"reply":0,"scanner":1,"sensors":0,"spindles":0,"state":0,"tools":0,"volChanges":[0,0],"volumes":0},"spindles":[{"current":0,"state":"unconfigured"},{"current":0,"state":"unconfigured"}],"state":{"atxPower":null,"currentTool":-1,"gpOut":[],"laserPwm":null,"msUpTime":101,"nextTool":-1,"previousTool":-1,"status":"off","time":"2021-04-13T20:58:38","upTime":83},"tools":[]},"next":0}
Apr 13 20:58:38 ender3 DuetControlServer[1585]:    System.InvalidOperationException: The requested operation requires an element of type 'Number', but the target element has type 'Array'.
Apr 13 20:58:38 ender3 DuetControlServer[1585]:    at System.Text.Json.JsonDocument.TryGetValue(Int32 index, Int32& value)
Apr 13 20:58:38 ender3 DuetControlServer[1585]:    at System.Text.Json.JsonElement.GetInt32()
Apr 13 20:58:38 ender3 DuetControlServer[1585]:    at DuetControlServer.Model.Updater.Run() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/Model/Updater.cs:line 157

root@ender3 ~ # dpkg -l duetcontrolserver
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name              Version      Architecture Description
+++-=================-============-============-============================================
ii  duetcontrolserver 3.3-b2       armhf        Control server application for Duet 3 series

I was hoping to do another batch of small sequential prints with the new firmware

[Serbitar]

FIRMWARE_NAME: RepRapFirmware for Duet 3 Mini 5+ FIRMWARE_VERSION: 3.3beta2+1 ELECTRONICS: Duet 3 Mini5plus WiFi FIRMWARE_DATE: 2021-04-09 14:40:21

This is the first version that causes the DCS log spam

[Serbitar]

Actually this is a bit weird. I just went through the remaining versions in sequence and 3.3b2+2 isn't spamming the logs.

[dc42]

@serbitar, I've updated the binary at https://www.dropbox.com/sh/0mwfe22wepk5efh/AAB6Lp0rwBPjVK2R4SxbKhLya?dl=0 to avoid DCS reporting those errors. The build time reported by M115 is now 21:29:10. The previous version was 18:55.

[Serbitar]

@dc42 said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

With the 18:55 version I had a print paused over night and the printer didn't have any VSSA or VIN troubles.
Print has now been resumed and will upgrade again later today after it finishes.

[dc42]

Thanks. I am 99% certain that my rewrite of the ADC driver has fixed this issue, so I will mark it as solved. Feel free to revert that if the problem recurs.

For others who have experienced this issue, thanks for your patience. The fix is in the Duet3Firmware_Mini5plus.uf2 file at https://www.dropbox.com/sh/0mwfe22wepk5efh/AAB6Lp0rwBPjVK2R4SxbKhLya?dl=0. The 3.3beta3 release will include this fix.

[Serbitar]

@dc42 Yep, looks good. No issues in the last 24h.
Many thanks for sorting this out!

[Fletcher]

@dc42 Running a print successfully as we speak, after 24hr with no issues. Thank you for your work sorting this out!

[droftarts]

Many thanks to everyone who helped resolve this issue, for your patience, persistence and providing vital testing and feedback. I hope you didn’t waste too much filament!

Ian