Software Reset whilst printing 3.2-1 Beta

jay_s_uk

I was running a print last night (a calibration cube) when all of a sudden the board reset.
Here is the output of the M122 directly afterwards. Seems it was a software reboot.
I am running 3.2-1 in standalone mode on a mini 5+ wifi.

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.2-beta1 running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 8ULVP-1D6P9-F65J0-401FQ-2A03Z-RJW1V
Used output buffers: 3 of 40 (11 max)
=== RTOS ===
Static ram: 85692
Dynamic ram: 127196 of which 692 recycled
Exception stack ram used: 224
Never used ram: 48340
Tasks: NETWORK(ready,161) HEAT(blocked,335) TMC(blocked,63) MAIN(running,457) IDLE(ready,20) AIN(blocked,267)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:00:12 ago, cause: software
Last software reset at 2020-09-28 22:18, reason: StuckInSpinLoop, GCodes spinning, available RAM 47868, slot 0
Software reset code 0x4083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0400f80f BFAR 0xe000ed38 SP 0x200075fc Task MAIN
Stack: 0002f3e3 00069fae a10f0000 3f800000 42fc4dd3 3edb6db7 b6376c00 40623446 3fde0224 44fa0001 44fa0001 00000000 447a0001 447a0001 3fe66668 3ca3d70a 43af028f 3f800000 3f800000 20000010 42fc4dd3 00000001 00000000 20002eb4 0002f3e3 20000ff0 00000001
Error status: 0x020
Supply voltage: min 0.0, current 12.5, max 12.6, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, read errors 0, write errors 1, ifcnt 22, reads 1784, writes 7, timeouts 0, DMA errors 0
Driver 1: position 0, standstill, read errors 0, write errors 1, ifcnt 22, reads 1752, writes 7, timeouts 32, DMA errors 0, failedOp 0x6f
Driver 2: position 0, standstill, read errors 0, write errors 1, ifcnt 22, reads 1781, writes 7, timeouts 3, DMA errors 0, failedOp 0x01
Driver 3: position 0, standstill, read errors 0, write errors 1, ifcnt 21, reads 1784, writes 7, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, read errors 0, write errors 1, ifcnt 24, reads 1784, writes 7, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Driver 7: position 0, assumed not present
Date/time: 2020-09-28 22:18:54
Slowest loop: 18.85ms; fastest: 0.10ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.9ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 11.86ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.24
WiFi MAC address f4:cf:a2:71:b0:50
WiFi Vcc 3.38, reset reason Turned on by main processor
WiFi flash size 2097152, free heap 26400
WiFi IP address 192.168.1.27
WiFi signal strength -91dBm, reconnections 0, sleep mode modem
Clock register ffffffff
Socket states: 0 0 0 0 0 0 0 0

dc42

Thanks for your report. The trace suggests that it was looping forever within the SD card driver. I've seen this happen before on the Mini 5+ (which uses a different SD card peripheral and driver compared to other Duets), especially at high SD card clock rates or when the SD card connection is unreliable. It's already on my list to improve the vendor-provided driver to implement a timeout and retry.

jay_s_uk

Just had this happen again while not printing (machine was idle after finishing a print).
I'm assuming this is due tot he same issue?

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.2-beta1 running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 8ULVP-1D6P9-F65J0-401FQ-2A03Z-RJW1V
Used output buffers: 3 of 40 (11 max)
=== RTOS ===
Static ram: 85692
Dynamic ram: 127196 of which 692 recycled
Exception stack ram used: 224
Never used ram: 48340
Tasks: NETWORK(ready,133) HEAT(blocked,335) TMC(blocked,63) MAIN(running,457) IDLE(ready,20) AIN(blocked,269)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:00:17 ago, cause: software
Last software reset at 2020-09-29 11:07, reason: StuckInSpinLoop, GCodes spinning, available RAM 47868, slot 1
Software reset code 0x4083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0400f80f BFAR 0xe000ed38 SP 0x200075fc Task MAIN
Stack: 0002f3e3 00069fb0 a10f0000 3f800000 405dc168 3edb6db7 b6376c00 3e000002 3c4ccccf 43480000 40a00001 00000000 3f800000 00000000 432130a4 3f000000 3f000000 3f800000 3f800000 20000011 405dc168 00000001 00000000 20002eb4 0002f3e3 20000ff0 00000001
Error status: 0x020
Supply voltage: min 0.0, current 12.5, max 12.5, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, read errors 0, write errors 1, ifcnt 30, reads 2520, writes 7, timeouts 0, DMA errors 0
Driver 1: position 0, standstill, read errors 0, write errors 1, ifcnt 30, reads 2520, writes 7, timeouts 0, DMA errors 0
Driver 2: position 0, standstill, read errors 0, write errors 1, ifcnt 32, reads 2520, writes 7, timeouts 0, DMA errors 0
Driver 3: position 0, standstill, read errors 0, write errors 1, ifcnt 31, reads 2520, writes 7, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, read errors 0, write errors 1, ifcnt 30, reads 2520, writes 7, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Driver 7: position 0, assumed not present
Date/time: 2020-09-29 11:08:23
Slowest loop: 19.05ms; fastest: 0.10ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.9ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 10.18ms; fastest: 0.00ms
Responder states: HTTP(1) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 2 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.24
WiFi MAC address f4:cf:a2:71:b0:50
WiFi Vcc 3.38, reset reason Turned on by main processor
WiFi flash size 2097152, free heap 24544
WiFi IP address 192.168.1.27
WiFi signal strength -89dBm, reconnections 0, sleep mode modem
Clock register ffffffff
Socket states: 0 2 0 0 0 0 0 0

dc42

Yes, that's the same reason. Probably caused by SD card access to read the daemon.g file.

I've done a fix, so I can give you new firmware to try out. I suggest you leave it idle and see if it happens again.

jay_s_uk

@dc42 the printer is free for testing now and as I have no jobs for it, I can leave it idle all day

dc42

You are welcome to try the firmware at https://www.dropbox.com/s/r65fl63ttvm6fh0/Duet3Firmware_Mini5plus.uf2?dl=0. Caution, I haven't tested it on a version 0.2 prototype, so far I only tested the slightly different build for the later prototype.

jay_s_uk

@dc42 i'll get that updated shortly.
Is there an IAP file available to allow updating through DWC yet?

dc42

Yes, https://www.dropbox.com/s/iw2wx23ta226b3c/Duet3_SDiap_Mini5plus.bin?dl=0.

oozeBot

@dc42 Wil this build work on the Duet 3 (non-mini) / RPi? I've experienced the same thing several times.. Thanks

T3P3Tony

@oozeBot no that build wont.

jay_s_uk

@dc42
I left the printer on all night and did not have a reset.
However, I was printing a file this morning and complete access to the SD card was lost.
DWC reported that the file being printed was unreadable with an error code 1.
I was then unable to home the machine etc. I also attempted to re-read the SD card using M21 but DWC reported the card as unusable.

M21
Error: M21: Cannot initialise SD card 0: Card is unusable
Failed to get file list
Target drive is unmounted

After that, DWC started to complain that it couldn't write to its json file. All that was left was to reset the machine, at which point the SD card was available again.
Below is an output from M122 when the SD card was unusable but I'm not sure it tells you much.

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ prototype v0.2 version 3.2-beta1+1 running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 8ULVP-1D6P9-F65J0-401FQ-2A03Z-RJW1V
Used output buffers: 3 of 40 (14 max)
=== RTOS ===
Static ram: 85976
Dynamic ram: 128920 of which 692 recycled
Exception stack ram used: 544
Never used ram: 46012
Tasks: NETWORK(ready,119) HEAT(blocked,302) TMC(blocked,69) MAIN(running,442) IDLE(ready,19) AIN(blocked,258)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 01:53:43 ago, cause: software
Last software reset at 2020-09-30 07:13, reason: User, GCodes spinning, available RAM 46036, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x04430000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 0x00
Supply voltage: min 0.0, current 12.5, max 12.7, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 14610, standstill, SG min/max 0/420, read errors 0, write errors 1, ifcnt 40, reads 52208, writes 26, timeouts 375, DMA errors 0, failedOp 0x71
Driver 1: position 14041, standstill, SG min/max 0/416, read errors 0, write errors 1, ifcnt 40, reads 51579, writes 26, timeouts 1004, DMA errors 0, failedOp 0x71
Driver 2: position 3160, standstill, SG min/max 14/216, read errors 0, write errors 1, ifcnt 40, reads 52081, writes 26, timeouts 502, DMA errors 0, failedOp 0x6f
Driver 3: position 0, standstill, SG min/max 12/226, read errors 0, write errors 1, ifcnt 40, reads 52582, writes 26, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, SG min/max 0/456, read errors 0, write errors 1, ifcnt 32, reads 52589, writes 20, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Driver 7: position 0, assumed not present
Date/time: 2020-09-30 09:07:05
Slowest loop: 1718.57ms; fastest: 0.10ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 9.1ms, write time 18.4ms, max retries 1
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 109, MaxWait: 3701413ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 15576, completed moves: 15576, StepErrors: 0, LaErrors: 0, Underruns: 0, 4  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 916.52ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.24
WiFi MAC address f4:cf:a2:71:b0:50
WiFi Vcc 3.38, reset reason Turned on by main processor
WiFi flash size 2097152, free heap 25856
WiFi IP address 192.168.1.27
WiFi signal strength -85dBm, reconnections 0, sleep mode modem
Clock register ffffffff
Socket states: 0 0 0 0 0 0 0 0

jay_s_uk

Just had it lose the SD card whilst sat idle as well. Popping the SD card out and putting it back in give the same errors as before. M21 also doesn't work.

jay_s_uk

had another timeout that seemed to happen somewhere around the 45 minute mark. I will revert back to the other previous build for the moment

dc42

Thanks, it looks like resetting the SDHC controller isn't enough to clear the error when it occurs.

To restore access to the SD card, did you need to power down, or was a software reset (or the reset button) enough?

jay_s_uk

Using the reset button was enough. I didn't try an emergency reset

jay_s_uk

@dc42 Is there another build you would like me to try?

jay_s_uk

I've switched to SBC mode at the moment.
I can't have the 12864 display enabled though as it just constantly resets

dc42

@jay_s_uk said in Software Reset whilst printing 3.2-1 Beta:

I've switched to SBC mode at the moment.
I can't have the 12864 display enabled though as it just constantly resets

If you think there is a problem with the 12864 display driver, please start a new thread about that.

Regarding the original problem, I have been assuming it is the internal SD card that you were referring to. Please confirm.

jay_s_uk

@dc42 yes, it was the internal sd card with the mini 5+ in standalone.
The reboots were becoming quite erratic, hence the switch to SBC

jay_s_uk

@dc42 I tried the updated firmware linked to in the other thread and managed to print a file, with the machine being left on after that.
On the second print, the SD card was lost again and the board did not reconnect.
I tried connecting to it via USB but I was not getting a response so I was unable to obtain an M122 output.
DWC also didn't connect and I got the HTTP rejected error.
Here is an M122 output after repowering the machine.

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ prototype v0.2 version 3.2-beta2 running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 8ULVP-1D6P9-F65J0-401FQ-2A03Z-RJW1V
Used output buffers: 3 of 40 (11 max)
=== RTOS ===
Static ram: 88664
Dynamic ram: 131140 of which 700 recycled
Exception stack ram used: 224
Never used ram: 41416
Tasks: NETWORK(ready,166) HEAT(blocked,335) TMC(blocked,69) MAIN(running,431) IDLE(ready,20) AIN(blocked,267)
Owned mutexes:
=== Platform ===
Last reset 00:00:40 ago, cause: power up
Last software reset at 2020-10-04 07:56, reason: StuckInSpinLoop, GCodes spinning, available RAM 40944, slot 1
Software reset code 0x4083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0443080f BFAR 0xe000ed38 SP 0x20002634 Task NETW
Stack: 000304eb 0006b34e 41090000 80000000 4064d0e5 41472759 b6376c00 3331bb4c 40000000 3f317200 b5ddea0e 388ab30c bb360b06 43570000 42c8c083 00000000 00000000 00000000 00000000 60000010 00000008 00000008 200039b8 00000000 000304eb 2002a720 00000000
Error status: 0x00
Supply voltage: min 0.0, current 12.6, max 12.6, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 11, reads 5783, writes 11, timeouts 0, DMA errors 0
Driver 1: position 0, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 11, reads 5783, writes 11, timeouts 0, DMA errors 0
Driver 2: position 0, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 11, reads 5782, writes 11, timeouts 0, DMA errors 0
Driver 3: position 0, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 11, reads 5782, writes 11, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 12, reads 5782, writes 12, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Driver 7: position 0, assumed not present
Date/time: 2020-10-04 09:55:14
Slowest loop: 4.25ms; fastest: 0.11ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 2.3ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 10.81ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.24
WiFi MAC address f4:cf:a2:71:b0:50
WiFi Vcc 3.38, reset reason Turned on by main processor
WiFi flash size 2097152, free heap 24544
WiFi IP address 192.168.1.56
WiFi signal strength -82dBm, reconnections 0, sleep mode modem
Clock register ffffffff
Socket states: 0 0 0 0 0 0 0 0