Reset during print Part 2 - Duet 3.1.1 / 3.2beta3 - Duet3+SBC
-
Hi,
This is a follow up to previous thread, https://forum.duet3d.com/topic/16774/reset-during-print-duet-3-3-1-1-sbc?_=1604781143331
Unfortunately due to some personal issues I had to shelve this project so printer has sat unused since June. I have got some more time now so have dusted things down and trying again.
So first change I have moved SBC (Pi4 4GB) and touch screen onto a 100W 5V power supply instead of previous USB brick, this should be way more power than needed and rules out any power supply issues to SBC. Everything else is powered by a 400W 12V supply (bed taking maximum of 240W), again no power supply issues with this are reported.
I moved everything onto a new branded microsd card, have had issues with those previously causing weird bugs.
Finally I switched over to unstable branch and installed 3.2beta3 (but issues are identical to what was occuring in 3.1.1 as well).
The print is a very simple rectangle design with a sloped lip that is to top off the printers electronic box. It is designed in OpenSCAD and passes all vector checks, I have rerendered and resliced using Cura in previous thread with no success. All other parts for the electronics box were done the same and printed fine on older Duet2 printer, this is the first part I am attempting on this printer.
The printer has sucessfully printed calibration cubes and a test print of a moon lamp from thingiverse. However on this print it always cuts out around layers 7-10 (of ~50) it is never in the same place twice (print head always in a different location) and it just simply stops and Duet3 resets (SBC remains running) as if I had pressed the reset button mid print (have checked nothing anywhere that could short the reset or any other part of board).
I have tried twice today, first with new power supply and second after updating to 3.2beta3, this is the M122 from 2nd test.
M122 === Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.2-beta3 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 4 of 40 (11 max) === RTOS === Static ram: 122124 Dynamic ram: 139224 of which 24 recycled Never used RAM 130820, free system stack 200 words Tasks: Linux(ready,109) HEAT(blocked,296) CanReceiv(blocked,948) CanSender(blocked,371) CanClock(blocked,352) TMC(blocked,54) MAIN(running,1215) IDLE(ready,19) Owned mutexes: HTTP(MAIN) === Platform === Last reset 00:07:52 ago, cause: watchdog Last software reset details not available Error status: 0x00 MCU temperature: min 31.6, current 32.0, max 33.2 Supply voltage: min 13.0, current 13.0, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0 Driver 0: position 0, standstill, reads 29696, writes 14 timeouts 0, SG min/max 0/0 Driver 1: position 0, standstill, reads 29697, writes 14 timeouts 0, SG min/max 0/0 Driver 2: position 0, standstill, reads 29697, writes 14 timeouts 0, SG min/max 0/0 Driver 3: position 0, standstill, reads 29698, writes 14 timeouts 0, SG min/max 0/0 Driver 4: position 0, standstill, reads 29699, writes 14 timeouts 0, SG min/max 0/0 Driver 5: position 0, standstill, reads 29703, writes 11 timeouts 0, SG min/max 0/0 Date/time: 2020-11-07 20:37:49 Slowest loop: 159.17ms; fastest: 0.15ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 37.5MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP* is doing "M122" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === CAN === Messages sent 1890, send timeouts 1890, longest wait 0ms for type 0, free CAN buffers 47 === SBC interface === State: 0, failed transfers: 0 Last transfer: 22ms ago RX/TX seq numbers: 11728/15416 SPI underruns 0, overruns 0 Number of disconnects: 0, IAP RAM available 0x20aa0 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.2.0-beta3 Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 32.45 -
I have now tried a fresh install of latest duetpi with same results.
From DCS log.
Nov 15 15:55:50 duet3 DuetControlServer[9198]: [warn] Lost connection to Duet (Timeout while waiting for transfer ready pin) Nov 15 15:55:50 duet3 DuetControlServer[9198]: [warn] Bad header checksum (expected 0x9393, got 0x79bc) Nov 15 15:55:51 duet3 DuetControlServer[9198]: [warn] Note: RepRapFirmware didn't receive valid data either (code 0x00000000) Nov 15 15:55:51 duet3 DuetControlServer[9198]: [info] Connection to Duet established Nov 15 15:55:51 duet3 DuetControlServer[9198]: [warn] Controller has been reset Nov 15 15:55:51 duet3 DuetControlServer[9198]: [info] Starting macro file config.g on channel Trigger Nov 15 15:55:51 duet3 DuetControlServer[9198]: [error] M550: Machine name must consist of the same letters and digits as configured by the Linux hostname Nov 15 15:55:51 duet3 DuetControlServer[9198]: [info] Aborted job file Nov 15 15:55:51 duet3 DuetControlServer[9198]: [warn] M307: Heater 0 appears to be over-powered. If left on at full power, its temperature is predicted to reach 365C Nov 15 15:55:52 duet3 DuetControlServer[9198]: [info] Starting macro file config-override.g on channel Trigger Nov 15 15:55:52 duet3 DuetControlServer[9198]: [info] Finished macro file config-override.g Nov 15 15:55:52 duet3 DuetControlServer[9198]: [warn] M307: Heater 0 appears to be over-powered. If left on at full power, its temperature is predicted to reach 226C Nov 15 15:55:52 duet3 DuetControlServer[9198]: [info] Finished macro file config.gM122
=== Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.2-beta3 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 3 of 40 (21 max) === RTOS === Static ram: 122124 Dynamic ram: 139224 of which 24 recycled Never used RAM 130820, free system stack 200 words Tasks: Linux(blocked,101) HEAT(blocked,296) CanReceiv(blocked,948) CanSender(blocked,371) CanClock(blocked,352) TMC(blocked,54) MAIN(running,1097) IDLE(ready,19) Owned mutexes: HTTP(MAIN) === Platform === Last reset 00:06:45 ago, cause: watchdog Last software reset at 2020-11-15 15:01, reason: User, Move spinning, available RAM 130780, slot 2 Software reset code 0x0004 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0xffffffff Task Linu Error status: 0x00 MCU temperature: min 31.4, current 31.5, max 32.3 Supply voltage: min 13.0, current 13.0, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0 Driver 0: position 0, standstill, reads 63565, writes 0 timeouts 0, SG min/max not available Driver 1: position 0, standstill, reads 63566, writes 0 timeouts 0, SG min/max not available Driver 2: position 0, standstill, reads 63565, writes 0 timeouts 0, SG min/max not available Driver 3: position 0, standstill, reads 63565, writes 0 timeouts 0, SG min/max not available Driver 4: position 0, standstill, reads 63565, writes 0 timeouts 0, SG min/max not available Driver 5: position 0, standstill, reads 63565, writes 0 timeouts 0, SG min/max not available Date/time: 2020-11-15 16:02:36 Slowest loop: 145.54ms; fastest: 0.24ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 37.5MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP* is doing "M122" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === CAN === Messages sent 1493, send timeouts 1493, longest wait 0ms for type 0, free CAN buffers 47 === SBC interface === State: 0, failed transfers: 0 Last transfer: 22ms ago RX/TX seq numbers: 54313/13286 SPI underruns 0, overruns 0 Number of disconnects: 0, IAP RAM available 0x20aa0 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.2.0-beta3 Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 32.82 -
You do not have RRF 3.2.0-b3.2 installed. Please run M997 again or upload the MB6HC binary on the System page and confirm the update prompt there.
-
@chrishamm apologies, didn't notice 3.2beta3.2 update until I posted this.
I have now updated to this and now for something different... Print got a lot further than previously (Layer 21) and then just froze, this time board did not reboot.
Full syslog at the time
Nov 15 17:42:57 duet3 DuetWebServer[523]: #033[40m#033[1m#033[33mwarn#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Server.Kestrel[22] Nov 15 17:42:59 duet3 DuetWebServer[523]: Heartbeat took longer than "00:00:01" at "11/15/2020 17:42:00 +00:00". This could be caused by thread pool starvation. Nov 15 17:43:53 duet3 DuetControlServer[434]: [info] System time has been changed Nov 15 17:44:48 duet3 DuetWebServer[523]: #033[40m#033[1m#033[33mwarn#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Server.Kestrel[22] Nov 15 17:44:57 duet3 DuetWebServer[523]: Heartbeat took longer than "00:00:01" at "11/15/2020 17:44:34 +00:00". This could be caused by thread pool starvation. Nov 15 17:45:32 duet3 DuetWebServer[523]: #033[40m#033[1m#033[33mwarn#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Server.Kestrel[22] Nov 15 17:45:41 duet3 DuetWebServer[523]: Heartbeat took longer than "00:00:01" at "11/15/2020 17:45:16 +00:00". This could be caused by thread pool starvation.Screen attached to pi has frozen and web connection sits on "Connecting".
Left while writing this post (to save filament waste I am using a modified version now with temps set to 0 so it goes through the motions without actually printing) and after a while it suddenly restarted printing.
Looking at logs (extrusion warning because of zeroed temps) it did last print move at 17:42:55 and then two at 17:45:53 and 17:45:58 (did think I heard it move but when I went to check display was still frozen so thought I imagined it) and then started reprinting fully (and still continuing now 10 mins later) at 17:51:01
The DCS section has nothing (other than the restarting transfer shortly before) - tool not driven warnings removed from all logs for clarity.
Nov 15 17:41:42 duet3 DuetControlServer[434]: [warn] Restarting transfer because a bad data response was received (0x00000005) Nov 15 17:41:51 duet3 DuetControlServer[434]: [warn] Restarting transfer because a bad header response was received (0x0003015f) Nov 15 17:42:54 duet3 DuetControlServer[434]: [info] System time has been changed Nov 15 17:43:53 duet3 DuetControlServer[434]: [info] System time has been changed Nov 15 17:45:51 duet3 DuetControlServer[434]: [info] System time has been changed Nov 15 17:45:52 duet3 DuetControlServer[434]: [info] System time has been changed Nov 15 17:46:46 duet3 DuetControlServer[434]: [info] System time has been changed Nov 15 17:48:03 duet3 DuetControlServer[434]: [info] System time has been changed Nov 15 17:50:52 duet3 DuetControlServer[434]: [info] System time has been changed Nov 15 17:50:52 duet3 DuetControlServer[434]: [info] System time has been changedEDIT: M122 taken shortly after it restarted printing...
=== Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.2-beta3.2 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (12 max) === RTOS === Static ram: 122236 Dynamic ram: 139232 of which 64 recycled Never used RAM 130660, free system stack 126 words Tasks: Linux(ready,67) HEAT(blocked,296) CanReceiv(blocked,948) CanSender(blocked,352) CanClock(blocked,352) TMC(blocked,20) MAIN(running,1073) IDLE(ready,19) Owned mutexes: HTTP(MAIN) === Platform === Last reset 01:44:39 ago, cause: software Last software reset at 2020-11-15 16:13, reason: User, GCodes spinning, available RAM 130700, slot 0 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0xffffffff Task Linu Error status: 0x00 MCU temperature: min 30.7, current 32.5, max 33.3 Supply voltage: min 12.9, current 13.0, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 11.9, current 12.0, max 12.1, under voltage events: 0 Driver 0: position 27914, ok, reads 37426, writes 19 timeouts 0, SG min/max 0/280 Driver 1: position 6439, ok, reads 37426, writes 19 timeouts 0, SG min/max 0/545 Driver 2: position 1480, standstill, reads 37438, writes 7 timeouts 0, SG min/max 0/317 Driver 3: position 0, standstill, reads 37439, writes 7 timeouts 0, SG min/max 0/332 Driver 4: position 0, standstill, reads 37427, writes 19 timeouts 0, SG min/max not available Driver 5: position 0, standstill, reads 37446, writes 0 timeouts 0, SG min/max not available Date/time: 2020-11-15 17:57:40 Slowest loop: 243.63ms; fastest: 0.09ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 37.5MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 372, MinFreeDm: 339, MaxWait: 249818ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 18341, completed moves 18327, StepErrors 0, LaErrors 0, Underruns [0, 0, 28], CDDA state 3 === AuxDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 1 Movement lock held by null HTTP* is doing "M122" in state(s) 0 Telnet is idle in state(s) 0 File* is doing "G0 F7200 X82.716003 Y80.648003" in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue* is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === CAN === Messages sent 24971, send timeouts 24971, longest wait 0ms for type 0, free CAN buffers 47 === SBC interface === State: 0, failed transfers: 2 Last transfer: 23ms ago RX/TX seq numbers: 52632/53918 SPI underruns 1, overruns 1 Number of disconnects: 8, IAP RAM available 0x20a30 Buffer RX/TX: 1168/1704-0 === Duet Control Server === Duet Control Server v3.2.0-beta3 File: Buffered code: G0 X152.632 Y162.681 Buffered code: G0 X152.632 Y162.84 Buffered code: G1 F4800 X174.699 Y80.486 E2.1268 Buffered code: G0 F7200 X174.463 Y80.487 Buffered code: G1 F4800 X214.597 Y120.621 E1.41584 Buffered code: G0 F7200 X214.599 Y120.621 Buffered code: G1 F4800 X221.858 Y118.676 E0.18746 Buffered code: G0 F7200 X221.858 Y117.7 Buffered code: G1 F4800 X184.646 Y80.487 E1.31277 Buffered code: G1 X184.54 Y80.381 Buffered code: G0 F7200 X184.54 Y80.648 Buffered code: G0 X185.88 Y80.648 Buffered code: G0 X185.88 Y80.485 Buffered code: G1 F4800 X163.813 Y162.84 E2.12682 Buffered code: G1 X163.775 Y162.984 Buffered code: G0 F7200 X163.775 Y162.681 Buffered code: G0 X165.175 Y162.681 Buffered code: G0 X165.175 Y162.84 Buffered code: G1 F4800 X82.822 Y80.487 E2.90522 Buffered code: G1 X82.716 Y80.381 Buffered code: G0 F7200 X82.716 Y80.648 Buffered code: G0 X81.524 Y80.648 Buffered code: G0 X81.524 Y80.486 Buffered code: G1 F4800 X59.457 Y162.84 E2.1268 Buffered code: G0 F7200 X59.457 Y162.681 Buffered code: G0 X58.26 Y162.681 Buffered code: G0 X58.26 Y162.84 Buffered code: G1 F4800 X58.139 Y162.719 E0.00427 Buffered code: G0 F7200 X58.139 Y162.544 Buffered code: G1 F4800 X213.707 Y120.86 E4.01755 Buffered code: G1 X213.851 Y120.822 Buffered code: G0 F7200 X213.78 Y120.656 ==> 1496 bytes Code buffer space: 2392 Configured SPI speed: 8000000 Hz Full transfers per second: 29.52 File /opt/dsf/sd/gcodes/[PLA] CFFFP_Top - zero temps.gcode is selected, processing -
Sorry for double posting but an update...
Print is still continuing (almost complete now judging by movements) however rpi gui is totally frozen still (mouse pointer not even moving).
Remote web panel working fine however print status shows as 100% complete on layer 21 (where it froze) status box at top shows busy and is updating correctly though with tool position (have also reloaded page).
-
If DCS reports many "system time has changed" messages then its internal update task stalled longer than 5 seconds every time. I suspect the underlying problem may be related to the DWS error messages, too, but TBH I have never observed those messages yet.
I noticed bad SD card performance may cause lags and if yours is damaged, those kinds of problems could be expected. I suggest you rewrite the latest DuetPi version to your board, update everything, and then change once again to the unstable branch. Please use the official Raspberry Pi Imager this time and let it verify the written content. When you start this tool, you can select custom third-party images at the bottom of the distro selection and then select the corresponding DuetPi img file.
If the same problem persists, maybe leave
htoprunning in a console in the background after a fresh start and check if a process is taking an excessive amount of CPU or RAM. -
Your issues seem similar to mine. I was getting a few different errors
Warning: Tool 0 was not driven because its heater temperatures were not high enough or it has a heater faultConnection interrupted, attempting to reconnect... DCS has been stoppedWarning: Lost connection to Duet (Board is not available (no data response))11/6/2020, 5:21:16 PM Cancelled printing file 0:/gcodes/a1.gcode, print time was 0h 19m 11/6/2020, 5:21:15 PM Warning: Controller has been resetRemoving the pi fixed the issue so I'm waiting until the communication issues are resolved before going back to the SPC
https://forum.duet3d.com/topic/19563/duet-3-sbc-dcs-has-stopped/ -
@chrishamm that was with the fresh install on a new (Sandisk Ultra) memory card. But I have redone to double check as previously did use Rufus to write. I have used RPi imager this time and verified. All updates done and rebooted, M997 done to confirm all updates OK and shown running 3.2beta3.2.
And this time back to reset ~1hr into print.
Kept a casual eye on htop during print, chromium did sit fairly high on CPU usage averaging ~55% but spiking as high as 98% this is without any plugins installed (did have Visualizer on previous) and only window open on gui is standard DWC.
Nothing telling on syslog (added M302 P1 to get rid of tool not driven warnings)
Nov 16 11:21:35 duet3 systemd[1]: Started Session c5 of user pi. Nov 16 11:54:33 duet3 DuetControlServer[8548]: [warn] Lost connection to Duet (Board is not available (no header)) Nov 16 11:54:54 duet3 DuetControlServer[8548]: [info] Connection to Duet established Nov 16 11:54:54 duet3 DuetControlServer[8548]: [warn] Controller has been reset Nov 16 11:54:54 duet3 DuetControlServer[8548]: [info] Starting macro file config.g on channel Trigger Nov 16 11:54:54 duet3 DuetControlServer[8548]: [warn] File: Out-of-order reply: 'G0/G1: insufficient axes homed' Nov 16 11:54:54 duet3 DuetControlServer[8548]: [error] G0/G1: insufficient axes homed Nov 16 11:54:54 duet3 DuetControlServer[8548]: [warn] File: Out-of-order reply: 'G0/G1: insufficient axes homed' Nov 16 11:54:54 duet3 DuetControlServer[8548]: [error] G0/G1: insufficient axes homed Nov 16 11:54:54 duet3 DuetControlServer[8548]: [warn] File: Out-of-order reply: 'G0/G1: insufficient axes homed' Nov 16 11:54:54 duet3 DuetControlServer[8548]: [error] G0/G1: insufficient axes homed Nov 16 11:54:54 duet3 DuetControlServer[8548]: [warn] File: Out-of-order reply: 'G0/G1: insufficient axes homed' Nov 16 11:54:54 duet3 DuetControlServer[8548]: [error] G0/G1: insufficient axes homed Nov 16 11:54:54 duet3 DuetControlServer[8548]: [info] Aborted job file Nov 16 11:54:55 duet3 DuetControlServer[8548]: [warn] M307: Heater 0 appears to be over-powered. If left on at full power, its temperature is predicted to reach 365C Nov 16 11:54:55 duet3 DuetControlServer[8548]: [info] Starting macro file config-override.g on channel Trigger Nov 16 11:54:55 duet3 DuetControlServer[8548]: [info] Finished macro file config-override.g Nov 16 11:54:55 duet3 DuetControlServer[8548]: [warn] M307: Heater 0 appears to be over-powered. If left on at full power, its temperature is predicted to reach 226C Nov 16 11:54:55 duet3 DuetControlServer[8548]: [info] Event logging stopped Nov 16 11:54:55 duet3 DuetControlServer[8548]: [info] Event logging to /opt/dsf/sd/sys/eventlog.txt started Nov 16 11:54:55 duet3 DuetControlServer[8548]: [info] Finished macro file config.gM122
=== Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.2-beta3.2 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (10 max) === RTOS === Static ram: 122236 Dynamic ram: 139232 of which 24 recycled Never used RAM 130700, free system stack 200 words Tasks: Linux(ready,81) HEAT(blocked,297) CanReceiv(blocked,948) CanSender(blocked,371) CanClock(blocked,354) TMC(blocked,54) MAIN(running,1217) IDLE(ready,19) Owned mutexes: HTTP(MAIN) === Platform === Last reset 00:02:40 ago, cause: watchdog Last software reset at 2020-11-15 23:19, reason: User, GCodes spinning, available RAM 130660, slot 2 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0xffffffff Task Linu Error status: 0x00 MCU temperature: min 35.0, current 36.8, max 36.9 Supply voltage: min 13.0, current 13.0, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0 Driver 0: position 0, standstill, reads 890, writes 14 timeouts 0, SG min/max 0/0 Driver 1: position 0, standstill, reads 891, writes 14 timeouts 0, SG min/max 0/0 Driver 2: position 0, standstill, reads 891, writes 14 timeouts 0, SG min/max 0/0 Driver 3: position 0, standstill, reads 892, writes 14 timeouts 0, SG min/max 0/0 Driver 4: position 0, standstill, reads 893, writes 14 timeouts 0, SG min/max 0/0 Driver 5: position 0, standstill, reads 897, writes 11 timeouts 0, SG min/max 0/0 Date/time: 2020-11-16 11:57:34 Slowest loop: 0.92ms; fastest: 0.10ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 37.5MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP* is doing "M122" in state(s) 0 Telnet is idle in state(s) 0 File* is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === CAN === Messages sent 640, send timeouts 640, longest wait 0ms for type 0, free CAN buffers 47 === SBC interface === State: 0, failed transfers: 0 Last transfer: 20ms ago RX/TX seq numbers: 61520/5305 SPI underruns 0, overruns 0 Number of disconnects: 0, IAP RAM available 0x20a30 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.2.0-beta3 Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 2.79 -
@dhusolo Just had a look through your thread, looks like that will be the way to go for now which is a shame as whole design of this printer is around the SBC and display design. Did you have any issues going over to standalone? I did try to do that a few days back but having problems getting board to start in standalone mode, most likely not got SD card right as my board only had the old duetpi image (but will do seperate thread on that later if I still have issues).
I am just redoing print now with debug mode as advised in that thread to see if it gives any further insight.
-
@Via Can you please share your G-code file? The other thread is not related to this issue and that one is likely fixed in the latest beta. Your M122 excerpt reveals that the Duet had an unexpected watchdog reset and it would really help if we manage to reproduce this.
-
@chrishamm original and zero temps version:
[PLA] CFFFP_Top - ReRender.gcode
[PLA] CFFFP_Top - ReRender - zero temps.gcodePrint using debug just stopped, attached is everything in scrollback buffer. Failure part below.
[debug] Waiting for finish of G1 F4800 X221.859 Y142.854 E0.51614 [debug] File: Sent G1 F4800 X215.386 Y162.84 E2.88069, remaining space 184, needed 56 [debug] File: Sent G0 F7200 X215.386 Y162.681, remaining space 136, needed 48 [debug] File: Sent G0 X216.503 Y162.681, remaining space 96, needed 40 [debug] File: Sent G0 X216.503 Y162.84, remaining space 56, needed 40 [debug] File: Sent G1 F4800 X221.859 Y142.854 E0.51614, remaining space 0, needed 56 [warn] Bad header checksum (expected 0x0000, got 0x1890) [warn] Bad header checksum (expected 0x0000, got 0x1890) [warn] Bad header checksum (expected 0x0000, got 0x1890) [warn] Restarting transfer because the number of maximum retries has been exceeded [debug] Cancelled G1 F4800 X221.859 Y142.854 E0.51614 [debug] Cancelled G0 X216.503 Y162.84 [debug] Cancelled G0 X216.503 Y162.681 [warn] Controller has been reset [debug] Cancelled G0 F7200 X215.386 Y162.681 [debug] Cancelled G1 F4800 X215.386 Y162.84 E2.88069 [debug] Cancelled G0 X112.148 Y162.84 [debug] Cancelled G0 X112.148 Y162.681 [debug] Cancelled G0 F7200 X221.858 Y133.072 [debug] Cancelled G1 F4800 X141.964 Y162.84 E2.10885 [debug] Cancelled G1 F4800 X110.759 Y162.841 E2.86914 [debug] Cancelled G0 F7200 X215.504 Y127.32 [debug] Cancelled G1 X141.926 Y162.984 [debug] Cancelled G1 F4800 X214.831 Y127.353 E0.91645 [debug] Cancelled G0 F7200 X205.323 Y162.84 [debug] Cancelled G0 F7200 X141.926 Y162.681 [debug] Cancelled G1 F4800 X205.204 Y162.84 E2.88069 [debug] Cancelled G1 F4800 X123.417 Y81.182 E1.68581 [warn] File: Out-of-order reply: '' [warn] File: Out-of-order reply: '' [debug] Cancelled G0 X58.3 Y98.673 [debug] Cancelled G0 X144.11 Y162.84 [debug] Cancelled G0 F7200 X58.3 Y100.269 [debug] Cancelled G1 X58.101 Y100.269 [debug] Cancelled G1 F4800 X62.453 Y81.182 E2.88069 [debug] Cancelled G1 F4800 X58.139 Y100.125 E0.48921 [debug] Cancelled G0 X144.11 Y162.681 [debug] Cancelled G0 F7200 X63.215 Y81.182 [debug] Cancelled G0 F7200 X133.729 Y81.182 [debug] Cancelled G0 X58.139 Y98.673 [debug] Cancelled G1 F4800 X134.028 Y81.181 E2.10885 [debug] Cancelled G0 F7200 X110.615 Y162.681 [debug] Cancelled G1 X110.615 Y162.879 [debug] Cancelled G0 F7200 X123.547 Y81.182 [debug] Cancelled G1 F4800 X221.859 Y133.675 E0.22419 [info] Aborted job file [info] Cancelled printing file 0:/gcodes/[PLA] CFFFP_Top - ReRender - zero temps.gcode, print time was 1h 18m [debug] Requesting update of key job, seq 47 -> 48 [debug] Updated key job -
@Via I flashed 3.1.1 to the board with Bossa and made a new SD card config. It didn't take too long
-
I've put Duet3 into standalone now, retried print using 3.1.1 and print failed again. M122 shows Memory protection fault at time of failure.
=== Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (11 max) === RTOS === Static ram: 154604 Dynamic ram: 162852 of which 44 recycled Exception stack ram used: 272 Never used ram: 75444 Tasks: NETWORK(ready,364) ETHERNET(blocked,436) HEAT(blocked,1200) CanReceiv(suspended,3820) CanSender(suspended,1488) CanClock(blocked,1452) TMC(blocked,204) MAIN(running,4472) IDLE(ready,76) Owned mutexes: === Platform === Last reset 00:00:47 ago, cause: software Last software reset at 2020-11-18 11:53, reason: Memory protection fault, spinning module GCodes, available RAM 75108 bytes (slot 0) Software reset code 0x4163 HFSR 0x00000000 CFSR 0x00000001 ICSR 0x04427804 BFAR 0x00000000 SP 0x204175c4 Task MAIN Stack: 0000000a 0000000a 40070000 20417668 204176b9 00000015 00000000 0000000d 0046207b 00000052 20417630 Error status: 0 MCU temperature: min 37.2, current 37.4, max 38.0 Supply voltage: min 13.0, current 13.0, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0 Driver 0: standstill, reads 38262, writes 14 timeouts 0, SG min/max 0/0 Driver 1: standstill, reads 38263, writes 14 timeouts 0, SG min/max 0/0 Driver 2: standstill, reads 38263, writes 14 timeouts 0, SG min/max 0/0 Driver 3: standstill, reads 38264, writes 14 timeouts 0, SG min/max 0/0 Driver 4: standstill, reads 38264, writes 14 timeouts 0, SG min/max 0/0 Driver 5: standstill, reads 38268, writes 11 timeouts 0, SG min/max 0/0 Date/time: 2020-11-18 11:54:07 Slowest loop: 6.14ms; fastest: 0.14ms === Storage === Free file entries: 10 SD card 0 detected, interface speed: 25.0MBytes/sec SD card longest read time 0.8ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === AuxDDARing === Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 9.70ms; fastest: 0.03ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions HTTP sessions: 1 of 8 - Ethernet - State: active Error counts: 0 0 0 0 0 Socket states: 5 2 2 2 2 0 0 0 === CAN === Messages sent 190, longest wait 0ms for type 0 === Linux interface === State: 0, failed transfers: 0 Last transfer: 47750ms ago RX/TX seq numbers: 0/1 SPI underruns 0, overruns 0 Number of disconnects: 0 Buffer RX/TX: 0/0-0 -
I've got more details about this problem, it looks like
lxpanel(a program for the desktop running in the background) has a memory leak and which, at some point, causes problems with DSF. I hope this will be fixed by Raspbian soon.@Via Please try out 3.2.0-b3.2 in standalone mode, the memory protection fault should be fixed in that build.
-
@chrishamm Thanks for the update.
Would using the non-gui version of duetpi and forgoing connected screen work around this issue for the short term keeping SBC control?
Have updated standalone to 3.2.0-b3.2 and running test again now.
-
@Via Yes, I think so. My printer has the GUI-less variant installed (DuetPi lite), so that probably explains why I couldn't observe this problem before.
-
@chrishamm No luck again on standalone with 3.2.0-beta3.2, this time "AssertionFailed"
I will have another try later with duetpi lite.
=== Diagnostics === RepRapFirmware for Duet 3 MB6HC version 3.2-beta3.2 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode) Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD Used output buffers: 1 of 40 (11 max) === RTOS === Static ram: 122236 Dynamic ram: 168580 of which 376 recycled Never used RAM 101000, free system stack 180 words Tasks: NETWORK(ready,165) ETHERNET(blocked,109) HEAT(blocked,293) CanReceiv(blocked,948) CanSender(blocked,371) CanClock(blocked,358) TMC(blocked,54) MAIN(running,1111) IDLE(ready,19) Owned mutexes: === Platform === Last reset 00:00:28 ago, cause: software Last software reset at 2020-11-19 11:11, reason: AssertionFailed, GCodes spinning, available RAM 101000, slot 0 Software reset code 0x4123 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00427000 BFAR 0x00000000 SP 0x2040fd54 Task MAIN Stack: 00000599 004886b0 00468ce5 00000000 ffffffff 20427700 2040e2a0 20427ab0 ffffffff 00000000 56a3ba63 a5a5a5a5 00468d9b 2040fda4 00000000 20423aa8 004664d5 20427aa0 00440cc3 00000000 20427aa4 20408001 2040fdb4 00000101 00469700 0046972e 61000000 Error status: 0x00 MCU temperature: min 37.1, current 37.8, max 38.0 Supply voltage: min 13.0, current 13.0, max 13.1, under voltage events: 0, over voltage events: 0, power good: yes 12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0 Driver 0: position 0, standstill, reads 9148, writes 14 timeouts 0, SG min/max 0/0 Driver 1: position 0, standstill, reads 9149, writes 14 timeouts 0, SG min/max 0/0 Driver 2: position 0, standstill, reads 9151, writes 14 timeouts 0, SG min/max 0/0 Driver 3: position 0, standstill, reads 9152, writes 14 timeouts 0, SG min/max 0/0 Driver 4: position 0, standstill, reads 9153, writes 14 timeouts 0, SG min/max 0/0 Driver 5: position 0, standstill, reads 9157, writes 11 timeouts 0, SG min/max 0/0 Date/time: 2020-11-19 11:12:08 Slowest loop: 6.68ms; fastest: 0.21ms === Storage === Free file entries: 10 SD card 0 detected, interface speed: 25.0MBytes/sec SD card longest read time 1.2ms, write time 0.0ms, max retries 0 === Move === Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms Bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 11.39ms; fastest: 0.03ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions HTTP sessions: 1 of 8 - Ethernet - State: active Error counts: 0 0 0 0 0 Socket states: 5 2 2 2 2 0 0 0 === CAN === Messages sent 113, send timeouts 113, longest wait 0ms for type 0, free CAN buffers 47 -
@Via , I will look into this later today.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 Many thanks.
Is it worth trying with DuetPi Lite? Was going to try earlier but work got in the way
-
@Via said in Reset during print Part 2 - Duet 3.1.1 / 3.2beta3 - Duet3+SBC:
@dc42 Many thanks.
Is it worth trying with DuetPi Lite? Was going to try earlier but work got in the way
I doubt it. The assertion failure looks like it was caused either by memory corruption, or possibly by a power brownout. I guess a hardware problem is also a possibility.
Can you set up a macro that runs the file in simulation mode within a loop, to see if you can provoke the failure that way?
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com
