Printer stops sporadically and then no longer responds

michaxy

Since a few days I have the problem that my printer after a few shifts simply stops and then also no longer responds.
The device is a HeVort, the mainboard is a Duet 3 MB6HC v1.01 currently in standalone mode.

So far I have tried different versions of the firmware, the problem started with v3.4.5 first in SBC mod, then without, then I tried 3.5.0 beta 3 and then went back down to 3.4.4, all without success.

M122 says the following:

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.4 (2022-10-20 16:19:01) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-9P63L-DJ3S0-7JTDD-3SS6N-9BLRB
Used output buffers: 3 of 40 (22 max)
=== RTOS ===
Static ram: 152824
Dynamic ram: 99552 of which 284 recycled
Never used RAM 95156, free system stack 208 words
Tasks: NETWORK(ready,29.8%,220) ETHERNET(notifyWait,0.2%,566) ACCEL(notifyWait,0.0%,348) HEAT(notifyWait,0.0%,322) Move(notifyWait,0.0%,351) CanReceiv(notifyWait,0.0%,943) CanSender(notifyWait,0.0%,335) CanClock(delaying,0.0%,334) TMC(notifyWait,8.1%,91) MAIN(running,61.8%,925) IDLE(ready,0.0%,30), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:11:23 ago, cause: software
Last software reset at 2023-04-18 19:56, reason: MemoryProtectionFault mmarValid daccViol, none spinning, available RAM 94328, slot 0
Software reset code 0x4172 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x00427804 BFAR 0x00000004 SP 0x20420050 Task NETW Freestk 549 ok
Stack: 40050000 000f423f 0000001e 204200cc 00000000 004526af 00481394 01010000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00489457 00000000 ffffffff 20418198 10000000 00000000 2041f400 ffffffff
Error status: 0x00
Step timer max interval 131
MCU temperature: min 26.4, current 26.7, max 38.1
Supply voltage: min 23.9, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.4, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/4/4, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 552, reads 2209, writes 14 timeouts 0
Driver 1: standstill, SG min 0, mspos 856, reads 2209, writes 14 timeouts 0
Driver 2: standstill, SG min 0, mspos 1016, reads 2209, writes 14 timeouts 0
Driver 3: standstill, SG min 0, mspos 744, reads 2209, writes 14 timeouts 0
Driver 4: standstill, SG min 0, mspos 232, reads 2209, writes 14 timeouts 0
Driver 5: standstill, SG min 0, mspos 1016, reads 2209, writes 14 timeouts 0
Date/time: 2023-04-18 20:08:12
Slowest loop: 3.43ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.2ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 6147, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 50), ts 3416/0/0
Tx timeouts 0,0,3415,0,0,2730 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 2.73ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 2 of 8
= Ethernet =
State: active
Error counts: 0 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

and my config-override.g it looks like this:

; config-override.g file generated in response to M500 at 2023-04-18 18:57
; This is a system-generated file - do not edit
; Heater model parameters
M307 H0 R0.693 K0.237:0.000 D2.90 E1.35 S1.00 B0
M307 H1 R5.751 K0.389:0.000 D5.80 E1.35 S1.00 B0 V23.8
; Workplace coordinates
G10 L2 P1 X0.00 Y0.00 Z0.00
G10 L2 P2 X0.00 Y0.00 Z0.00
G10 L2 P3 X0.00 Y0.00 Z0.00
G10 L2 P4 X0.00 Y0.00 Z0.00
G10 L2 P5 X0.00 Y0.00 Z0.00
G10 L2 P6 X0.00 Y0.00 Z0.00
G10 L2 P7 X0.00 Y0.00 Z0.00
G10 L2 P8 X0.00 Y0.00 Z0.00
G10 L2 P9 X0.00 Y0.00 Z0.00

Right now I'm a little stumped as to where else to look.

Phaedrux

Is that M122 report from after it has stopped responding?

Can you go through this and let us know what you see when it stops?

https://docs.duet3d.com/en/User_manual/Troubleshooting/Duet_wont_respond

Does it always come back up after a power cycle? If so, gather an M122 as soon as you've powered back up and share it here.

When it's not responding, are you able to connect by usb terminal and send commands?

michaxy

I got the report after the stop, the stop sounds like an emergency stop.

Yesterday i try also an m98 p"config.g" but i got an http time out .

Phaedrux

@michaxy said in Printer stops sporadically and then no longer responds:

Yesterday i try also an m98 p"config.g" but i got an http time out .

You may need to send that from USB terminal. It sounds like DWC lost connection when it re-enabled the network, as that command just runs the config.g as a macro.

michaxy

I haven't printed any more now, so I haven't had another stop yet. I have gone through the points in the link, the Led`s light up and flash as described under normal operation. Contact via USB with YAT was also possible.
(M122 sent and get response)

In the config I just changed M552 from static to dynamic IP, since then M98 p "config" gives me an output.

m98 p"config.g"
HTTP is enabled on port 80
FTP is disabled
TELNET is disabled
Warning: Heater 0 predicted maximum temperature at full power is 246°C
Warning: Heater 1 predicted maximum temperature at full power is 760°C

Next I would try another slicer and print an older file, maybe these stops come from the gcode.

Phaedrux

You can run a dry print with no material loaded and see if you can duplicate it.

If it stops again, use the usb terminal to gather an M122.

It's strange that it happens regardless of firmware version. So it could be a hardware issue or the gcode being printed.

michaxy

I hope its just the gcode.

michaxy

After 4 minutes with an older file, the printer stopped again. Like a reset, heaters off and the coordinates are all at zero.
But this time I was able to move the axes afterwards.

Very strange

M122

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.4 (2022-10-20 16:19:01) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-9P63L-DJ3S0-7JTDD-3SS6N-9BLRB
Used output buffers: 3 of 40 (13 max)
=== RTOS ===
Static ram: 152824
Dynamic ram: 99520 of which 112 recycled
Never used RAM 98216, free system stack 219 words
Tasks: NETWORK(ready,29.6%,220) ETHERNET(notifyWait,0.2%,561) ACCEL(notifyWait,0.0%,348) HEAT(notifyWait,0.0%,328) Move(notifyWait,0.0%,351) CanReceiv(notifyWait,0.0%,943) CanSender(notifyWait,0.0%,335) CanClock(delaying,0.0%,340) TMC(notifyWait,8.0%,91) MAIN(running,61.8%,925) IDLE(ready,0.4%,30), total 100.0%
Owned mutexes: LwipCore(NETWORK)
=== Platform ===
Last reset 00:00:26 ago, cause: software
Last software reset at 2023-04-19 21:00, reason: MemoryProtectionFault iaccViol, none spinning, available RAM 94316, slot 0
Software reset code 0x4172 HFSR 0x00000000 CFSR 0x00000001 ICSR 0x00427804 BFAR 0x00000000 SP 0x2041fe58 Task NETW Freestk 423 ok
Stack: 20427740 004a03af 00000061 20427743 004a03b3 0040d0a9 804002ea 810f0000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ffffffff 00000000 20427920 004a0434
Error status: 0x00
Step timer max interval 126
MCU temperature: min 23.3, current 23.9, max 34.4
Supply voltage: min 23.9, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.2, current 12.2, max 12.4, under voltage events: 0
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 632, reads 13489, writes 14 timeouts 0
Driver 1: standstill, SG min 0, mspos 584, reads 13490, writes 14 timeouts 0
Driver 2: standstill, SG min 0, mspos 696, reads 13490, writes 14 timeouts 0
Driver 3: standstill, SG min 0, mspos 696, reads 13490, writes 14 timeouts 0
Driver 4: standstill, SG min 0, mspos 696, reads 13490, writes 14 timeouts 0
Driver 5: standstill, SG min 0, mspos 232, reads 13490, writes 14 timeouts 0
Date/time: 2023-04-19 21:00:40
Slowest loop: 3.25ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.1ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 234, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 50), ts 131/0/0
Tx timeouts 0,0,130,0,0,102 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 2.55ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 2 of 8
= Ethernet =
State: active
Error counts: 0 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

dc42

@michaxy thanks for the M122 report. The machine reset due to an instruction access violation. This may have been caused by electrostatic discharge, a hardware fault, power surge, or possibly by a short-lived loss of power.

Electrostatic discharge (ESD) often occurs when plastic is extruded from a hot end that is electrically isolated. This builds up static charge on the hot end metalwork, which subsequently flashes over to the heater or temperature sensor connections, which feed it into the Duet.

To avoid this possible cause of unexpected resets, connect the hot end metalwork to Duet ground.

michaxy

Ok, then I will pull an additional ground wire tomorrow, test and report back. Does the cable have to be connected to the ground connection of the board or would the connection to the ground of the printer also be ok ?
I have mounted a rail with terminal blocks for phase, neutral and ground at the bottom of the control cabinet.

michaxy

Ground is connected and I was able to print the file from yesterday completely. For days again a sense of success.

Thanks for the help

dc42

@michaxy I'm glad that appears to have solved it. Usually you would connect mains ground to Duet ground too. The exception would be if you were using the Duet USB connector and wanted to avoid creating a ground loop.

michaxy

Rejoiced too soon, I tried to print a certain file again today and it reset again, despite grounding from the hot end.

I can not get rid of the feeling that it is the GCode, last time I took another file, it worked.

dc42

@michaxy if it always stops at the same place in that file then it's likely that RRF is having trouble with the GCode at that point. If it stops randomly than ESD or some other type of transient is likely to be the problem.

I suggest you add a connection between Duet VIN- and the mains ground that you connected the hot end metalwork to. I normally make this connection at the PSU by linking the mains ground terminal to one of the negative output terminals.

michaxy

The hot-end is connected to the GND connector at the power-in on the mainboard

michaxy

So, I deleted the firmware and reinstalled again, currently a print has been running for an hour, without stopping.
I assume in my up and downgrade attempts, I have screwed up something. I will continue to monitor the matter and report.

gumby

@michaxy I was having similar mystery problems at first, and it was always associated with a low voltage error.

Could it be a power supply issue? Do you have a spare you could test it with?

Cheers,
Gumby

michaxy

@gumby
I have no spare, unfortunately, voltages are monitored and from a voltage drop of 0.5 volts, the print is paused.
With me, however, everything has reset and partly I had to turn the printer off and on again, so that I can operate the again.

michaxy

Hello, sorry that I have not been in touch for so long, but family and work have left me little time for hobbies.

For whatever reason, I had commented out the acceleration sensor during the last firmware upgrade or downgrade (probably because the firmware version did not yet support the sensor) and since then it has been quiet, it has printed as if nothing had ever happened.
A few months ago I planned further upgrades, such as a conversion from 2 to 4 drives etc. I also added a Raspberry PI 5 . First I wanted to put the Pi into operation with firmware V 3.5.2 and test a few things and what can I say, it happened again. I started printing and after a few layers the device stopped.
The error message was “SPI connection has been reset”, M122 gave me a similar diagnosis as previously posted in this topic.
What I noticed was that a “daccViol” or “iaccViol” was repeatedly set in line 13 of the diagnosis.
I then downgraded to stand alone V3.5.2, same error pattern.
Finally I commented out the acceleration sensor and disconnected it, since then it works again.

Conclusion: either my wiring of the sensor is faulty or the sensor itself.

droftarts

@michaxy Can you post your config.g and response to M122? You say you have a 6HC, and that you disabled "the acceleration sensor". Do you mean the accelerometer? Are you running a 1LC toolboard as well (if so, which version), or external accelerometer?

Ian