Reset during print Duet 3, 3.1.1 + SBC
-
Have my new Duet3 build wired up and working, done a test calibration cube which printed fine. However on longer prints the printer is failing after an hour or so (this is second occurence in a row now, hoped first was one off glitch).
02/06/2020, 19:42:57 Warning: Lost connection to Duet (Timeout while waiting for transfer ready pin) 02/06/2020, 18:17:07 M32 "0:/gcodes/[PLA ] CFFFP_Top - No Text.gcode" File 0:/gcodes/[PLA ] CFFFP_Top - No Text.gcode selected for printing
Running 3.1.1, M122 info below
M122 === Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (10 max) === RTOS === Static ram: 154604 Dynamic ram: 162972 of which 56 recycled Exception stack ram used: 224 Never used ram: 75360 Tasks: NETWORK(ready,1972) HEAT(blocked,1188) CanReceiv(suspended,3820) CanSender(suspended,1488) CanClock(blocked,1436) TMC(blocked,204) MAIN(running,4944) IDLE(ready,76) Owned mutexes: === Platform === Last reset 00:47:22 ago, cause: watchdog Last software reset at 2020-06-02 18:09, reason: User, spinning module LinuxInterface, available RAM 75056 bytes (slot 2) Software reset code 0x0010 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0444a000 BFAR 0x00000000 SP 0xffffffff Task MAIN Error status: 0 MCU temperature: min 35.5, current 35.7, max 40.4 Supply voltage: min 13.0, current 13.0, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0 Driver 0: standstill, reads 62000, writes 14 timeouts 0, SG min/max 0/0 Driver 1: standstill, reads 62000, writes 14 timeouts 0, SG min/max 0/0 Driver 2: standstill, reads 62001, writes 14 timeouts 0, SG min/max 0/0 Driver 3: standstill, reads 62001, writes 14 timeouts 0, SG min/max 0/0 Driver 4: standstill, reads 62001, writes 14 timeouts 0, SG min/max 0/0 Driver 5: standstill, reads 62005, writes 11 timeouts 0, SG min/max 0/0 Date/time: 2020-06-02 20:30:19 Slowest loop: 4.05ms; fastest: 0.14ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 37.5MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === AuxDDARing === Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP* is ready with "M122" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon* is idle in state(s) 0 0, running macro Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 0.68ms; fastest: 0.01ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions HTTP sessions: 0 of 8 - Ethernet - State: disabled Error counts: 0 0 0 0 0 Socket states: 0 0 0 0 0 0 0 0 === CAN === Messages sent 11366, longest wait 0ms for type 0 === Linux interface === State: 0, failed transfers: 0 Last transfer: 19ms ago RX/TX seq numbers: 11820/26640 SPI underruns 0, overruns 0 Number of disconnects: 0 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.1.1 Daemon: Finishing macro daemon.g, started by system > Next stack level Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 32.32
-
Can anyone advise regarding this?
Happens without fail every time ~ 1-2 hours into a print. Tried to enable logging to see if that would give any further information but does not want to stay turned on...
2020-06-03 16:22:33 Event logging started 2020-06-03 16:22:33 Event logging stopped 2020-06-03 17:35:36 Event logging started 2020-06-03 17:35:36 Event logging stopped
(using M929 P"eventlog.txt" S1 in either config.g or manually in console just gives started/stopped message in eventlog)
-
Thanks for your report. Would you be able to share your gcode file that causes a reset?
-
[PLA+] CFFFP_Top - No Text.gcode
(Hoping that worked)
Very simple print. Not sure if it is the GCode causing as it does not reset at same point each time.
-
I see you are using Cura 4.5. Can you try updating to 4.6.1?
-
Updated to 4.6.1 and reprinting now... will see what happens in an hour or so.
-
No difference...
04/06/2020, 11:01:26 Warning: Controller has been reset 04/06/2020, 11:01:26 Connection to Duet established 04/06/2020, 11:01:26 Warning: Lost connection to Duet (Board is not available (no header))
M122 shows a different reset reason this time (Memory protection fault)...
M122 === Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (11 max) === RTOS === Static ram: 154604 Dynamic ram: 163060 of which 44 recycled Exception stack ram used: 224 Never used ram: 75284 Tasks: NETWORK(ready,1972) HEAT(blocked,1188) CanReceiv(suspended,3820) CanSender(suspended,1488) CanClock(blocked,1436) TMC(blocked,204) MAIN(running,4976) IDLE(ready,76) Owned mutexes: === Platform === Last reset 00:04:32 ago, cause: software Last software reset at 2020-06-04 11:01, reason: Memory protection fault, spinning module GCodes, available RAM 74900 bytes (slot 0) Software reset code 0x4163 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x04427804 BFAR 0x00000004 SP 0x20412984 Task NETW Stack: 00413cd3 00414280 61000000 e000ed04 10000000 20415abc 004142c5 a5a5a5a5 00462b3d a5a5a5a5 a5a5a5a5 Error status: 0 MCU temperature: min 31.1, current 31.2, max 34.9 Supply voltage: min 13.0, current 13.1, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0 Driver 0: standstill, reads 19525, writes 14 timeouts 0, SG min/max 0/0 Driver 1: standstill, reads 19526, writes 14 timeouts 0, SG min/max 0/0 Driver 2: standstill, reads 19526, writes 14 timeouts 0, SG min/max 0/0 Driver 3: standstill, reads 19527, writes 14 timeouts 0, SG min/max 0/0 Driver 4: standstill, reads 19527, writes 14 timeouts 0, SG min/max 0/0 Driver 5: standstill, reads 19531, writes 11 timeouts 0, SG min/max 0/0 Date/time: 2020-06-04 11:05:58 Slowest loop: 3.97ms; fastest: 0.14ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 37.5MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === AuxDDARing === Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP* is ready with "M122" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon* is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 1.10ms; fastest: 0.01ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions HTTP sessions: 0 of 8 - Ethernet - State: disabled Error counts: 0 0 0 0 0 Socket states: 0 0 0 0 0 0 0 0 === CAN === Messages sent 1088, longest wait 0ms for type 0 === Linux interface === State: 0, failed transfers: 0 Last transfer: 17ms ago RX/TX seq numbers: 44432/8840 SPI underruns 0, overruns 0 Number of disconnects: 0 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.1.1 Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 32.42
-
Thanks, a memory protection fault should be easier to trace. I'm on to it.
-
PS - one more question: do you use G29 S1 to load an existing height map before you print? If so, were any of the probe points skipped when it was generated? We are aware of an issue with G29 S1 + SBC when points were skipped, although we haven't known it cause these symptoms before.
-
@dc42 said in Reset during print Duet 3, 3.1.1 + SBC:
PS - one more question: do you use G29 S1 to load an existing height map before you print? If so, were any of the probe points skipped when it was generated? We are aware of an issue with G29 S1 + SBC when points were skipped, although we haven't known it cause these symptoms before.
I did, not aware of any skipped points though and heightmap appears ok...
RepRapFirmware height map file v2 generated at 2020-06-02 13:47 xmin,xmax,ymin,ymax,radius,xspacing,yspacing,xnum,ynum 10.00,270.00,20.00,220.00,-1.00,40.00,40.00,7,6 -0.967, -0.822, -0.710, -0.715, -0.767, -0.720, -0.560 -0.455, -0.522, -0.470, -0.395, -0.355, -0.435, -0.435 -0.285, -0.175, -0.182, -0.260, -0.255, -0.102, -0.205 0.078, 0.062, 0.062, 0.190, 0.125, -0.007, 0.065 0.218, 0.253, 0.420, 0.345, 0.335, 0.340, 0.370 0.495, 0.525, 0.663, 0.723, 0.755, 0.678, 0.673
-
Retried today without loading height map and doing probe prior to print (G30 + G29).
Same result, this time cutting out 44min into print. Memory protection fault again.
M122 === Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (10 max) === RTOS === Static ram: 154604 Dynamic ram: 163060 of which 44 recycled Exception stack ram used: 544 Never used ram: 74964 Tasks: NETWORK(ready,1972) HEAT(blocked,1188) CanReceiv(suspended,3820) CanSender(suspended,1384) CanClock(blocked,1436) TMC(blocked,68) MAIN(running,2868) IDLE(ready,76) Owned mutexes: === Platform === Last reset 13:38:50 ago, cause: power up Last software reset at 2020-06-04 11:01, reason: Memory protection fault, spinning module GCodes, available RAM 74900 bytes (slot 0) Software reset code 0x4163 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x04427804 BFAR 0x00000004 SP 0x20412984 Task NETW Stack: 00413cd3 00414280 61000000 e000ed04 10000000 20415abc 004142c5 a5a5a5a5 00462b3d a5a5a5a5 a5a5a5a5 Error status: 0 MCU temperature: min 20.8, current 30.3, max 34.4 Supply voltage: min 12.7, current 13.0, max 13.3, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 11.6, current 12.0, max 12.3, under voltage events: 0 Driver 0: standstill, reads 35687, writes 39 timeouts 0, SG min/max 0/781 Driver 1: standstill, reads 35692, writes 35 timeouts 0, SG min/max 0/600 Driver 2: standstill, reads 35688, writes 39 timeouts 0, SG min/max 0/1023 Driver 3: standstill, reads 35689, writes 39 timeouts 0, SG min/max 0/1023 Driver 4: standstill, reads 35681, writes 47 timeouts 0, SG min/max 0/807 Driver 5: standstill, reads 35718, writes 11 timeouts 0, SG min/max 0/0 Date/time: 2020-06-06 11:37:26 Slowest loop: 12.18ms; fastest: 0.15ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 37.5MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 269, MaxWait: 17378550ms Bed compensation in use: mesh, comp offset 0.000 === MainDDARing === Scheduled moves: 8671, completed moves: 8671, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === AuxDDARing === Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP* is ready with "M122" in state(s) 0 Telnet is idle in state(s) 0 File* is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue* is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon* is idle in state(s) 0 0, running macro Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 1.71ms; fastest: 0.01ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions HTTP sessions: 0 of 8 - Ethernet - State: disabled Error counts: 0 0 0 0 0 Socket states: 0 0 0 0 0 0 0 0 === CAN === Messages sent 196621, longest wait 0ms for type 0 === Linux interface === State: 0, failed transfers: 2 Last transfer: 18ms ago RX/TX seq numbers: 16465/2082 SPI underruns 2, overruns 2 Number of disconnects: 0 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.1.1 Daemon: Finishing macro daemon.g, started by system > Next stack level Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 9.97
-
@Via said in Reset during print Duet 3, 3.1.1 + SBC:
Retried today without loading height map and doing probe prior to print (G30 + G29).
Same result, this time cutting out 44min into print. Memory protection fault again.
M122 === Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (10 max) === RTOS === Static ram: 154604 Dynamic ram: 163060 of which 44 recycled Exception stack ram used: 544 Never used ram: 74964 Tasks: NETWORK(ready,1972) HEAT(blocked,1188) CanReceiv(suspended,3820) CanSender(suspended,1384) CanClock(blocked,1436) TMC(blocked,68) MAIN(running,2868) IDLE(ready,76) Owned mutexes: === Platform === Last reset 13:38:50 ago, cause: power up Last software reset at 2020-06-04 11:01, reason: Memory protection fault, spinning module GCodes, available RAM 74900 bytes (slot 0)
Thanks for the report. I'm sorry that something is causing your prints to fail.
The reset data says the last reset was "Power up" and happened over 13 hours ago. The memory protection fault is Thursday's report - look at the "Last software reset" date/time.
So this time the Duet hadn't crashed, it was idle for some reason. It looks like DSF had stopped sending it data. I'll ask @chrishamm whether the Linux Interface part of the diagnostics sheds any light on why.
BTW the memory protection fault makes little sense. The only explanations I can think of are a transient or power brownout upsetting the processor, a hardware fault in the processor, or a very obscure software bug - although it would be something of a miracle for a bug to trigger this precise memory protection fault.
-
@dc42 Thanks for reply, sorry didn't pay attention to the times on there I just looked at last reset reason.
As a test I did a 6 hour print yesterday of a random thingiverse download (moon lamp) which I thought would be good as a torture test, this printed completely with no issues.
So going back to this part I checked and verified OK in Openscad and re-rendered and sliced again, exactly same settings as yesterdays test with same filament and temperatures. This has just failed again 1 hour into print (never fails at same point) this time M122 (below) shows last reset reason as watchdog (and time matches with failure this time).
I am at a total loss, the Raspberry Pi (4B) is powered seperately with a 3amp supply I have another one so will swap out with that tomorrow just to see if that makes any difference.
If I can provide any further diagnostics please let me know.
M122 === Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (13 max) === RTOS === Static ram: 154604 Dynamic ram: 163060 of which 44 recycled Exception stack ram used: 360 Never used ram: 75148 Tasks: NETWORK(ready,1972) HEAT(blocked,1188) CanReceiv(suspended,3820) CanSender(suspended,1488) CanClock(blocked,1436) TMC(blocked,68) MAIN(running,4512) IDLE(ready,76) Owned mutexes: === Platform === Last reset 00:11:26 ago, cause: watchdog Last software reset at 2020-06-06 13:56, reason: User, spinning module LinuxInterface, available RAM 75132 bytes (slot 3) Software reset code 0x0010 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0444a000 BFAR 0x00000000 SP 0xffffffff Task MAIN Error status: 0 MCU temperature: min 30.4, current 30.6, max 32.8 Supply voltage: min 13.0, current 13.1, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0 Driver 0: standstill, reads 22931, writes 0 timeouts 0, SG min/max not available Driver 1: standstill, reads 22932, writes 0 timeouts 0, SG min/max not available Driver 2: standstill, reads 22929, writes 2 timeouts 0, SG min/max not available Driver 3: standstill, reads 22929, writes 2 timeouts 0, SG min/max not available Driver 4: standstill, reads 22931, writes 0 timeouts 0, SG min/max not available Driver 5: standstill, reads 22932, writes 0 timeouts 0, SG min/max not available Date/time: 2020-06-07 22:51:20 Slowest loop: 4.04ms; fastest: 0.21ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 37.5MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves: 1, completed moves: 1, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === AuxDDARing === Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP* is ready with "M122" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon* is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 0.47ms; fastest: 0.01ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions HTTP sessions: 0 of 8 - Ethernet - State: disabled Error counts: 0 0 0 0 0 Socket states: 0 0 0 0 0 0 0 0 === CAN === Messages sent 2117, longest wait 0ms for type 0 === Linux interface === State: 0, failed transfers: 0 Last transfer: 19ms ago RX/TX seq numbers: 58505/22212 SPI underruns 0, overruns 0 Number of disconnects: 0 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.1.1 Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 32.26