Duet3 freezes

caviara

Good day,

First of all i want to apologize for silly questions i may have, but i'm new to RR FW and Duet ecosystem in general.

I have a little (or not so little) problem with my Duet3 6ch board:
Approximately after hour of work - print stops completely in one place and cools down and in un-homed position according to DWC .
When this happens PanelDue 5i stops responding to touch, but still shows values and restart button on it doesn't solve the issue.

What i did to try to resolve:

decrease the speed with every next print (100> 60> 40 mm/s)
done a meshmap of the bed (read somewhere that this may be one possible reason)
On the 3rd run i have tried to start login by using:

M929 P"05062020_3rdrun.txt" S1

but from DWC i cannot see such file so would appreciate if someone tell me how to get it.

result from M122 just after the last stall:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKD2-3SD6R-1U56F
Used output buffers: 1 of 40 (20 max)
=== RTOS ===
Static ram: 154604
Dynamic ram: 162968 of which 44 recycled
Exception stack ram used: 272
Never used ram: 75328
Tasks: NETWORK(ready,300) ETHERNET(blocked,436) HEAT(blocked,1176) CanReceiv(suspended,3820) CanSender(suspended,1488) CanClock(blocked,1424) TMC(blocked,192) MAIN(running,4528) IDLE(ready,76)
Owned mutexes:
=== Platform ===
Last reset 00:08:49 ago, cause: power up
Last software reset at 2020-06-05 14:12, reason: User, spinning module GCodes, available RAM 75312 bytes (slot 1)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0444a000 BFAR 0x00000000 SP 0xffffffff Task MAIN
Error status: 0
MCU temperature: min 41.4, current 42.3, max 43.7
Supply voltage: min 23.9, current 23.9, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0
Driver 0: standstill, reads 46311, writes 14 timeouts 0, SG min/max 0/0
Driver 1: standstill, reads 46312, writes 14 timeouts 0, SG min/max 0/0
Driver 2: standstill, reads 46313, writes 14 timeouts 0, SG min/max 0/0
Driver 3: standstill, reads 46313, writes 14 timeouts 0, SG min/max 0/0
Driver 4: standstill, reads 46314, writes 14 timeouts 0, SG min/max 0/0
Driver 5: standstill, reads 46314, writes 14 timeouts 0, SG min/max 0/0
Date/time: 2020-06-05 16:27:10
Slowest loop: 7.35ms; fastest: 0.21ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.9ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = 2 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 5.85ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 2 of 8
- Ethernet -
State: active
Error counts: 0 0 0 0 0
Socket states: 5 2 2 2 2 2 0 2
=== CAN ===
Messages sent 2115, longest wait 0ms for type 0
=== Linux interface ===
State: 0, failed transfers: 0
Last transfer: 529082ms ago
RX/TX seq numbers: 0/1
SPI underruns 0, overruns 0
Number of disconnects: 0
Buffer RX/TX: 0/0-0

any suggestions will be appreciated

jay_s_uk

The mesh bed levelling issue is only present when the duet is attached to a raspberry pi.

Can you also post the gcode file thats causing you an issue? Or is this not restricted to a particular gcode file?

caviara

@jay_s_uk said in Duet3 freezes:

The mesh bed levelling issue is only present when the duet is attached to a raspberry pi.

Completely forgot to mention - the board is in standalone mode.
@jay_s_uk said in Duet3 freezes:

Can you also post the gcode file thats causing you an issue? Or is this not restricted to a particular gcode file?

Last two of the files attached - tried only with this one for the moment as i want to print a case for my board.

Edit: it is stopping approximately at 4th layer
40mm_Duet3_MB_Case_NR.gcode 40mm_Duet3_MB_Case.gcode

Phaedrux

Are you actively cooling the board?

caviara

@Phaedrux
At the moment - no, but i touched all major components when it happened for second time and they seems "OK"...
This was my thought at the second freeze...
...because on my first machine (old i3 clone) i had overheating issues when i first installed TMC2310, but they were way much hotter than all the components on Duet now (burn finger within less than a second).
Also i cannot understand why display stops responding to the commands, but still shows all information on the main screen correctly.

Can measure the temperatures if you advice to do so

Phaedrux

What firmware version is your paneldue using? Should be 1.24.

Can you post your full config.g?

Do you get any error messages in the gcode console of the DWC or paneldue?

caviara

@Phaedrux
my PanelDue FW is 1.23.2

no error messages on the display or DWC upon freeze.

current config files:
config.g config-override.g

and because i did power-cycle of the machine (decided to go to bed and then i saw your answer) on boot i discovered the log file in the System part of DWC:
05062020_3rdrun.txt
but it seems to be empty

Phaedrux

https://duet3d.dozuki.com/Wiki/PanelDue_Firmware_update
Update the paneldue firmware, though I'm not sure it will make much of a change as the improvements for Duet 3 were for when a Pi was connected, not standalone. Still..

If you get another crash like this please capture another M122.

caviara

@Phaedrux
Updated PanelDue to 1.24 as suggested - same behavior when problem occurs. I did managed to notice that on the problem display restarted.
This time after hitting reset on the display (which didn't made it operational as it was before) i did disconnected and reconnected it back and it worked (I was able to home the machine while the nossle was still hot).

Worth mentioning that i disabled completely the second extruder for this test in config.g - because i don't have motor connected there yet.

Here is the M122 reading:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKD2-3SD6R-1U56F
Used output buffers: 3 of 40 (11 max)
=== RTOS ===
Static ram: 154604
Dynamic ram: 162968 of which 44 recycled
Exception stack ram used: 528
Never used ram: 75072
Tasks: NETWORK(ready,276) ETHERNET(blocked,436) HEAT(blocked,1112) CanReceiv(suspended,3820) CanSender(suspended,1420) CanClock(blocked,1424) TMC(blocked,68) MAIN(running,4480) IDLE(ready,76)
Owned mutexes:
=== Platform ===
Last reset 00:03:38 ago, cause: power up
Last software reset at 2020-06-06 13:22, reason: User, spinning module GCodes, available RAM 75328 bytes (slot 2)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0444a000 BFAR 0x00000000 SP 0xffffffff Task MAIN
Error status: 0
MCU temperature: min 41.8, current 43.6, max 44.3
Supply voltage: min 23.9, current 23.9, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0
Driver 0: standstill, reads 20415, writes 21 timeouts 0, SG min/max 0/148
Driver 1: standstill, reads 20417, writes 21 timeouts 0, SG min/max 0/158
Driver 2: standstill, reads 20417, writes 21 timeouts 0, SG min/max 0/372
Driver 3: standstill, reads 20418, writes 21 timeouts 0, SG min/max 0/443
Driver 4: standstill, reads 20425, writes 14 timeouts 0, SG min/max 0/0
Driver 5: standstill, reads 20429, writes 11 timeouts 0, SG min/max 0/0
Date/time: 2020-06-06 14:27:52
Slowest loop: 6.52ms; fastest: 0.21ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 3.9ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 375, MinFreeDm: 373, MaxWait: 140163ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 12, completed moves: 12, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = 2 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 6.38ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 0 0 0
Socket states: 5 2 2 2 2 2 0 2
=== CAN ===
Messages sent 877, longest wait 0ms for type 0
=== Linux interface ===
State: 0, failed transfers: 0
Last transfer: 218079ms ago
RX/TX seq numbers: 0/1
SPI underruns 0, overruns 0
Number of disconnects: 0
Buffer RX/TX: 0/0-0

I was monitoring machine closely this time and measured the temperatures on the drivers an CPU. E0, X, Y and CPU both read around 36deg on the top ceramic part Z was cooler to touch and I dint measured it. Also all the voltage regulators were a bit cooler to touch.

Here is the result of the M996 command:
06062020_5thrun.txt

One thing that bother me at this point is:
can it be a PS problem?

Reason for the question:
When it happened i immediately looked at the DWC and saw main voltage reading increased 5>14>23.9V ...
it was in less than half a second though so i didn't manage to see the other reading for 12V.

P.S.
and while on the Power-Supply theme a question:
I saw your Z-Bot images and wonder - shall i ground all the motors and frame to the PS and Duet? I mean can this be a problem that i didn't do it?

dc42

@caviara said in Duet3 freezes:

Last reset 00:03:38 ago, cause: power up

Did you power the machine down and up three and a half minutes before you ran the M122 command? If you didn't, you have a power problem.

I saw your Z-Bot images and wonder - shall i ground all the motors and frame to the PS and Duet?

We recommend that you ground the frame and motors because it helps avoid build up of static charge. If the motors are screwed to a metal frame, you need only ground the frame.

caviara

@dc42 said in Duet3 freezes:

Did you power the machine down and up three and a half minutes before you ran the M122 command? If you didn't, you have a power problem.

No i did not...

Yesterday i was trying to capture the power-supply on my USB scope, but as you can expect - it did fail shortly after my laptop battery depleted
I think to repeat this procedure tomorrow and hopefully this time will be able to catch the spike (if one)

Today i did a quick "stress test" for the PS (bed heated to 110 and hotend to 260) and i think i did get a power drop but it did not reflected on the Duet.

In the mean time think to purchase new PS - to be on a safe side.

PS
Thanks for the advise on the grounding - will do it. For sure it will not hurt (especially considering that i use IGUS bearings everywhere).

caviara

Good morning!

Just wanted to finish this thread by confirming that it is (was) a power-supply issue.
yesterday night after almost 3 hours printing i was able to capture the problem:

It was a cheap Chinese brand 480W power-supply and the time increased from 1 to 3 hours because i did some re-soldering for the transistors inside to get them align better to the heat sink (in this case aluminum block connected to the chassis).
Anyways i received yesterday brand new MeanWell power-suply and hope that it will finish my waste of filament for good.

Thank you for your help!

dc42

I'm glad you found the problem!

I used a no-name Chinese PSU in my delta printer. A few weeks ago a burning smell started coming from the printer, even though the printer was still running normally. I opened up the PSU and found this:

alt text

One toroidal inductor and one resistor badly burned. The PCB was completely scorched under the inductor.

I replaced it with a Meanwell.