Duet Maestro random reboot (Hard Fault) during large prints

dc42

This is a very strange fault. I can tell that the processor was about to execute a network transfer or had just executed one at the tie of the transfer, however the stack pointer register appears to have been corrupted so the stack trace makes very little sense. Please provide some more information:

You said " Since the thermistor connector for the first hotend was not working as expected on the first Maestro and which I thought could be the problem for the stops, I replaced it with a new one, which fixed the thermistor problem, but not the stops." Are you saying that you have used 2 different Duet Maestro boards, and they both had this problem (longer prints stopping)?
If you run the same print more than once, does it always stop at exactly the same place, or is there a randomness in how long it prints for before it stops?
Please provide your config.g file, and also provide a GCode file that shows the problem. I will run it on the bench and see if I can replicate the problem. Unfortunately, file uploads to this forum don't seem to work properly at present, but you can host the files on Dropbox or Google drive and link to them in your post.
Please run another print (without filament if you like) and after it stops, run M122 and post the results again. I want to see if the reset happened at exactly the same place again.

Gizzle

@dc42 hey there, thank you for the answer.

Regarding your questions:

It happened on both boards, yes.
I currently print the gcode a second time, but I need to wait for the staff of my professor to give me informations since I am not at the lab today
Asked the staff for the files and will immediately send them to you
See 2

droftarts

@Gizzle Also, how are you producing the gcode, ie which slicer, or is it custom generated?

Ian

Gizzle

@droftarts Was generated with Cura.

Gizzle

Ok, I think I found the problem. I forgot to mention that I had extruder skips. Today I replaced the stock extruder with a Bondtech including a pancake stepper, reduced the current from 1.6A (yes, very high but not done by me) to the maximum of 0.8A, which the stepper is rated for, and now the printer seems to work again. Need to do further tests next week, but I am very confident.

So my last question is: Could a high temperature of the driver caused by the high amperage lead to a system shutoff?

dc42

@Gizzle said in Duet Maestro random reboot (Hard Fault) during large prints:

So my last question is: Could a high temperature of the driver caused by the high amperage lead to a system shutoff?

It certainly shouldn't! If the drivers get too hot, warning messages should appear on the GCode Console of Duet Web Control. Did you see any? This worked last time I tested it. If the drivers go on heating up, they will eventually shut down, which should give rise to a different error message on the console.

Gizzle

Unfortunately not. Wasn't connected to the printer during the print. But I'll print a new print and and report back...

Gizzle

@dc42 The problem occured today again. Unfortunately the guys in the lab send me an empty log file. But I have the config and a gcode which stopped.

https://www.dropbox.com/s/7ib6btuo0yumo46/CFFFP_chibigrim-twiesner-3.gcode?dl=0

https://www.dropbox.com/s/m29xw2cnosvt7os/download.zip?dl=0

dc42

I'm sorry, if I am to look into this then I need the M122 report.

Gizzle

@dc42 Somehow my phone didn't show the log properly, so here it is:
12.12.2019, 07:12:04: Connected to 141.75.30.240
12.12.2019, 07:12:38: M122: === Diagnostics ===
RepRapFirmware for Duet 2 Maestro version 2.04 running on Duet Maestro 1.0
Board ID: 08DJM-956DU-LL3T0-6J9FL-3S86Q-9T2LP
Used output buffers: 3 of 24 (7 max)
=== RTOS ===
Static ram: 19772
Dynamic ram: 86864 of which 124 recycled
Exception stack ram used: 264
Never used ram: 24048
Tasks: NETWORK(ready,764) HEAT(blocked,1296) MAIN(running,3904) IDLE(ready,200)
Owned mutexes:
=== Platform ===
Last reset 16:48:54 ago, cause: software
Last software reset at 2019-12-11 14:23, reason: Hard fault, spinning module Platform, available RAM 23724 bytes (slot 0)
Software reset code 0x4030 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x04427803 BFAR 0xe000ed38 SP 0x20000ff4 Task 0x5754454e
Stack: 004057a9 0045c534 20000000 2000847c e000ed01 20001c0c a5a5a5a5 a5a5a5a5 00000001 20008380 bcfbb5af 00000000 004093e5 a5a5a5a5 10000000 e000ed04 20001c0c a5a5a5a5 00409447 a5a5a5a5 00449765 a5a5a5a5 00ffffff
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 15.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 25.3, current 25.6, max 30.6
Supply voltage: min 0.0, current 24.3, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, read errors 0, write errors 1, ifcount 55, reads 62059, timeouts 0
Driver 1: standstill, read errors 0, write errors 1, ifcount 55, reads 62059, timeouts 0
Driver 2: standstill, read errors 0, write errors 1, ifcount 59, reads 62059, timeouts 0
Driver 3: standstill, read errors 0, write errors 1, ifcount 45, reads 62059, timeouts 0
Driver 4: standstill, read errors 0, write errors 1, ifcount 27, reads 62059, timeouts 0
Driver 5: ok, read errors 0, write errors 0, ifcount 0, reads 0, timeouts 62065
Driver 6: ok, read errors 0, write errors 0, ifcount 0, reads 0, timeouts 62065
Date/time: 2019-12-12 07:12:37
Slowest loop: 3.49ms; fastest: 0.05ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 160, MinFreeDm: 160, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0
=== Heat ===
Bed heaters = 0, chamberHeaters = -1 -1
=== GCodes ===
Segments left: 0
Stack records: 1 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
lcd is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 4.87ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
Interface state 5, link 100Mbps full duplex

dc42

Thanks. Looking at both M122 reports, both crashed at a nonsensical point, suggesting either hardware failure or stack corruption. But both were in the network subsystem at the time. On inspecting the code, I realised that if the W5500 network chip returned a larger value for the amount of data it had than is theoretically possible, that would cause DMA to overflow a buffer, which might lead to stack corruption. There is a chance that this was responsible for both crashes. I have added a check in the 2.05 release.

Gizzle

@dc42 Thank you for your response. I got the info from my professor that sometimes our network has some small outtages. Could these be the reason for the problem? I started a print without connected network today to check if the print finishes...

dc42

@Gizzle said in Duet Maestro random reboot (Hard Fault) during large prints:

@dc42 Thank you for your response. I got the info from my professor that sometimes our network has some small outtages. Could these be the reason for the problem? I started a print without connected network today to check if the print finishes...

Not unless there is a bug in the W5500 chip that causes it to return faulty info when there is a network outage.

I have released firmware 2.05, so please upgrade to that and see if it helps.

Gizzle

@dc42 will do that the next time I'm at the printer. Thank you very much for your hard work!

Gizzle

@dc42 Updated today to the 2.05 release. The print without ethernet from last week went fine, the one from today on 2.05 and over ethernet aborted again.
Here is the M122 output:

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 Maestro version 2.05 running on Duet Maestro 1.0
Board ID: 08DJM-956DU-LL3T0-6J9FL-3S86Q-9T2LP
Used output buffers: 1 of 24 (20 max)
=== RTOS ===
Static ram: 19804
Dynamic ram: 87256 of which 124 recycled
Exception stack ram used: 204
Never used ram: 23684
Tasks: NETWORK(ready,900) HEAT(blocked,1296) MAIN(running,3896) IDLE(ready,160)
Owned mutexes:
=== Platform ===
Last reset 00:03:31 ago, cause: software
Last software reset at 2019-12-18 10:25, reason: Assertion failed, spinning module Platform, available RAM 23440 bytes (slot 1)
Software reset code 0x4090 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x04427000 BFAR 0xe000ed38 SP 0x20000f84 Task 0x5754454e
Stack: 00000f3c 00463a24 0044a949 00000001 200084a8 00000001 00000000 00000000 0044c2e3 200084a8 2000075c 00000000 0044c641 00000000 00000000 2000849c 20000fd8 200084a8 2000075c 20001000 00000000 a5a5a5a5 a5a5a5a5
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 15.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 30.8, current 30.9, max 32.3
Supply voltage: min 0.0, current 24.2, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, read errors 0, write errors 1, ifcount 27, reads 8457, timeouts 0
Driver 1: standstill, read errors 0, write errors 1, ifcount 27, reads 8457, timeouts 0
Driver 2: standstill, read errors 0, write errors 1, ifcount 27, reads 8457, timeouts 0
Driver 3: standstill, read errors 0, write errors 1, ifcount 25, reads 8457, timeouts 0
Driver 4: standstill, read errors 0, write errors 1, ifcount 20, reads 8457, timeouts 0
Driver 5: ok, read errors 0, write errors 0, ifcount 0, reads 0, timeouts 8463
Driver 6: ok, read errors 0, write errors 0, ifcount 0, reads 0, timeouts 8463
Date/time: 2019-12-18 10:29:10
Slowest loop: 1.07ms; fastest: 0.05ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 160, MinFreeDm: 160, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0
=== Heat ===
Bed heaters = 0, chamberHeaters = -1 -1
=== GCodes ===
Segments left: 0
Stack records: 1 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
lcd is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 7.19ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(1) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
Interface state 5, link 100Mbps full duplex

dc42

That's another strange failure. As the cause may be a hardware failure, I think the time has come to try another Maestro. Please ask your supplier to replace your Maestro under warranty, and tag the one you return with a link to this thread.

Gizzle

@dc42 That's funny, because that's the second maestro I am using right now...
That's why I can't really believe that a second one has also the same symptoms...

dc42

@Gizzle said in Duet Maestro random reboot (Hard Fault) during large prints:

@dc42 That's funny, because that's the second maestro I am using right now...
That's why I can't really believe that a second one has also the same symptoms...

Just to confirm: you have the same fault on 2 different Maestro boards? That makes a difference.

Gizzle

@dc42 Yes, this is the case

dc42

@Gizzle said in Duet Maestro random reboot (Hard Fault) during large prints:

Updated today to the 2.05 release. The print without ethernet from last week went fine, the one from today on 2.05 and over ethernet aborted again.

When you say "over Ethernet", do you mean that you uploaded the file to SD card over Ethernet, and then printed form SD card?

We've seen a very small number of instances of file corruption when uploading large files over Ethernet. So I am wondering whether file corruption could be the issue - although a corrupt file shouldn't cause the firmware to crash, unless the firmware file itself was corrupted when you uploaded it (RRF3 checks for firmware corruption, but RRF2 doesn't). Here are some suggestions:

Re-run the file that crashed, in simulation mode, without uploading it again, and see whether the simulation completed successfully.
Use DWC 2.0.4 and enable the feature to do CRC checking when it uploads files.
Run one of those prints again, and after it has printed several layers, run M122 and post the results here.

You could also consider switching to RepRapFirmware 3.0RC1 in case it either solves the problem or provides more information, although that would require changes to your config.g file.