DAMNED CRASH - Stuck in spin loop

gate546 · 13 Jun 2018, 12:07

@dc42
After two days of a very long print, and only at 10 minutes from the end .......
I state that the machine worked normally for a long time, this was the first print after the firmware update.
Apparently everything seems normal .....
HELP ......

=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.0(RTOS) running on Duet WiFi 1.0 or 1.01
Board ID: 08D6M-91AST-L23S4-7JTD2-3S46P-94W1J
Used output buffers: 3 of 20 (14 max)
=== RTOS ===
Static ram: 28380
Dynamic ram: 95776 of which 0 recycled
Exception stack ram used: 348
Never used ram: 6568
Task NETWORK ready, free stack 324
Task HEAT blocked, free stack 1148
Task MAIN running, free stack 3560
=== Platform ===
Last reset 00:15:16 ago, cause: power up

Last software reset at 2018-06-13 11:34, reason: Stuck in spin loop, spinning module GCodes, available RAM 6248 bytes (slot 0)
Software reset code 0x4043 HFSR 0x00000000, CFSR 0x00000000, ICSR 0x0441f80f, BFAR 0xe000ed38, SP 0x2001ff94
Stack: 0043e245 00000000 00444a57 00000000 00445e2d 004590ac 20006c58 20002a00 00000000 22240000 44480000 11200000 2001d29c 00000000 0001955c 0000001c 00444c1b 0000c0d6 ffffffed 00000000 00f00000 e000ef34 c0000000 00000003

Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms
MCU temperature: min 16.4, current 26.6, max 26.8
Supply voltage: min 12.0, current 12.1, max 12.4, under voltage events: 0, over voltage events: 0
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2018-06-13 13:52:48
Slowest loop: 239.35ms; fastest: 0.07ms
=== Move ===
Hiccups: 0, StepErrors: 0, LaErrors: 0, FreeDm: 240, MinFreeDm 236, MaxWait: 2915221410ms, Underruns: 0, 0
Scheduled moves: 9, completed moves: 9
Bed compensation in use: none
Bed probe heights: 0.000 0.000 0.000 0.000 0.000
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.1
Heater 1 is on, I-accum = 0.2
=== GCodes ===
Segments left: 0
Stack records: 1 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0

dc42 · 13 Jun 2018, 16:40

I'm sorry to hear that you had a reset so close to the end of a long print.

I have examined the stack trace, but unfortunately I can't tell anything from it because for this type of reset (stuck in spin loop), I need a longer stack trace. I will amend the stack trace in the 2.01 release to provide more information.

The most likely causes of the reset that I can think of are:

An electrical transient causing the board to malfunction. Can you confirm that you did not have a USB connection to the Duet at the time? Were any large items of electrical equipment turned on or off around the time of the reset?
A failure of the SD card interface to either return the requested data or time out. I have seen evidence of this happening in the past, but not recently. I will change the stack trace information to record whether an SD card timeout had been logged at the time of the reset.
An undiagnosed firmware issue.

gate546 · 13 Jun 2018, 16:59

Hello,

thank you for your quick answer...:

@dc42 said in DAMNED CRASH - Stuck in spin loop:

I'm sorry to hear that you had a reset so close to the end of a long print.

I have examined the stack trace, but unfortunately I can't tell anything from it because for this type of reset (stuck in spin loop), I need a longer stack trace. I will amend the stack trace in the 2.01 release to provide more information.

The most likely causes of the reset that I can think of are:

An electrical transient causing the board to malfunction. Can you confirm that you did not have a USB connection to the Duet at the time? Were any large items of electrical equipment turned on or off around the time of the reset?

This is not possible because the printer have a dedicated UPS, that is also filtered

A failure of the SD card interface to either return the requested data or time out. I have seen evidence of this happening in the past, but not recently. I will change the stack trace information to record whether an SD card timeout had been logged at the time of the reset.

This is at least unlikely because for a long time it never happened

An undiagnosed firmware issue.

I do not know if it can help you, but maybe it could be a problem on the WEB Server part, the only thing I did was to connect with a browser (local) to see the progress
the printer was printing autonomously and never connected before.
Hoping that this will help your troubleshooting
Cheers
Dommy

(just started again the long print crossing the finger...)

timcurtis67 · 13 Jun 2018, 17:29

@gate546 said in DAMNED CRASH - Stuck in spin loop:

A failure of the SD card interface to either return the requested data or time out. I have seen evidence of this happening in the past, but not recently. I will change the stack trace information to record whether an SD card timeout had been logged at the time of the reset.

This is at least unlikely because for a long time it never happened

I ran into a similar issue last week with a few failures (2 during random prints) near the end of long prints. I replaced the SD card with a new one and the issue went away.

I tried re-formatting the old card but the problem still came back after a few good prints. So I went to a new SD card and had no problems after that.

I never had an issue with my SD card until I started having an issue with my SD. Sometimes they work until they don't work anymore.

gate546 · 13 Jun 2018, 17:56

@timcurtis67 said in DAMNED CRASH - Stuck in spin loop:

@gate546 said in DAMNED CRASH - Stuck in spin loop:

A failure of the SD card interface to either return the requested data or time out. I have seen evidence of this happening in the past, but not recently. I will change the stack trace information to record whether an SD card timeout had been logged at the time of the reset.

This is at least unlikely because for a long time it never happened

I ran into a similar issue last week with a few failures (2 during random prints) near the end of long prints. I replaced the SD card with a new one and the issue went away.

I tried re-formatting the old card but the problem still came back after a few good prints. So I went to a new SD card and had no problems after that.

I never had an issue with my SD card until I started having an issue with my SD. Sometimes they work until they don't work anymore.

Thank you to share...
Dommy

denke · 13 Jun 2018, 19:08

@dc42
Isn't it possible to manually craft a resurrect file to continue such a print?
If any additional information is needed to craft one, cannot it be added to the dump?

(Maybe even auto create a resurrect file on a post-crash bootup?)

timothyz · 13 Jun 2018, 19:23

@gate546
I had a similar issue, it was resolved by reformatting with a larger allocation unit size (the largest possible). that resolved a number of issues and made a number of things work better.

-Tim

Phaedrux · 13 Jun 2018, 19:29

@gate546 said in DAMNED CRASH - Stuck in spin loop:

This is not possible because the printer have a dedicated UPS, that is also filtered

I used to use a UPS on my old printer, but I stopped when one day the printer suddenly stopped and restarted in the middle of a print while I was sitting next to it. I looked over at the UPS and saw that it was doing an automatic self-test. The self-test is not supposed to interrupt power but I think due to the high load of the bed heater the voltage drooped enough to reset the controller.

Just a thought.

dc42 · 13 Jun 2018, 19:49

@gate546 said in DAMNED CRASH - Stuck in spin loop:

I do not know if it can help you, but maybe it could be a problem on the WEB Server part, the only thing I did was to connect with a browser (local) to see the progress
the printer was printing autonomously and never connected before.

Did the reset occur immediately after you connected the web browser, or some time later?

gate546 · 13 Jun 2018, 19:54

This post is deleted!

gate546 · 13 Jun 2018, 19:55

@dc42 said in DAMNED CRASH - Stuck in spin loop:

@gate546 said in DAMNED CRASH - Stuck in spin loop:

I do not know if it can help you, but maybe it could be a problem on the WEB Server part, the only thing I did was to connect with a browser (local) to see the progress
the printer was printing autonomously and never connected before.

Did the reset occur immediately after you connected the web browser, or some time later?

Immediately, or better, immediately as soon as have launched the browser, in fact the browser was not connected, remain on connecting status then I look the printer and I see that the printer was stopped, crashed, without any apparent problem, only stopped.

dc42 · 13 Jun 2018, 20:08

Thanks, it does sound that connecting the browser may have triggered the problem. I'll do some more tests on connecting a browser while a print is in progress.

gate546 · 13 Jun 2018, 20:22

@gate546 said in DAMNED CRASH - Stuck in spin loop:

@dc42 said in DAMNED CRASH - Stuck in spin loop:

@gate546 said in DAMNED CRASH - Stuck in spin loop:

I do not know if it can help you, but maybe it could be a problem on the WEB Server part, the only thing I did was to connect with a browser (local) to see the progress
the printer was printing autonomously and never connected before.

Did the reset occur immediately after you connected the web browser, or some time later?

Immediately, or better, immediately as soon as have launched the browser, in fact the browser was not connected, remain on connecting status then I look the printer and I see that the printer was stopped, crashed, without any apparent problem, only stopped.

One more info, the involved gcode file is over a normal SD card that is installed on the 7" Panel and NOT on the micro SD Card that is inside the Duet Wifi board

Phaedrux · 13 Jun 2018, 20:24

@gate546 How long was the cable between the panel and the Duet? What type of cabling is used?

gate546 · 13 Jun 2018, 20:43

@dc42 said in DAMNED CRASH - Stuck in spin loop:

Thanks, it does sound that connecting the browser may have triggered the problem. I'll do some more tests on connecting a browser while a print is in progress.

this sound goods....
I hope that you are able to replicate the issue.

gate546 · 13 Jun 2018, 20:44

@phaedrux said in DAMNED CRASH - Stuck in spin loop:

@gate546 How long was the cable between the panel and the Duet? What type of cabling is used?

Hello,
the TFT panel and the board are very close, 25 cm. of wire, 1 inch.
the cable is a normal quadripolar cable, single wires, with connectors on both sides.
However we need to remember a point, this printer worked from a couple of years without any problems, and I also have printed very long files without difficulty.

This thing happened only now, in conjunction with the firmware change.
Since the new firmware is RTOS, then with separate tasks, I'm trying to provide DC42 as much information as possible because, maybe, they are useful for help it for troubleshooting this issue.

Phaedrux · 13 Jun 2018, 20:57

@gate546 Yes, of course. Merely chasing down possibilities.

dc42 · 14 Jun 2018, 12:40

@gate546 said in DAMNED CRASH - Stuck in spin loop:

One more info, the involved gcode file is over a normal SD card that is installed on the 7" Panel and NOT on the micro SD Card that is inside the Duet Wifi board

That may be very relevant. The SPI link to the SD card on the PanelDue doesn't have the same hardware timeout mechanisms as the interface to the micro SD card. Also if the ribbon cable is too long then the signal integrity may be degraded. For best reliability, I recommend printing from the micro SD card.

Do you have any other SPI-connected devices, for example a PT100 or thermocouple daughter board?

gate546 · 14 Jun 2018, 12:58

@dc42 said in DAMNED CRASH - Stuck in spin loop:

@gate546 said in DAMNED CRASH - Stuck in spin loop:

One more info, the involved gcode file is over a normal SD card that is installed on the 7" Panel and NOT on the micro SD Card that is inside the Duet Wifi board

That may be very relevant. The SPI link to the SD card on the PanelDue doesn't have the same hardware timeout mechanisms as the interface to the micro SD card. Also if the ribbon cable is too long then the signal integrity may be degraded. For best reliability, I recommend printing from the micro SD card.

Do you have any other SPI-connected devices, for example a PT100 or thermocouple daughter board?

@dc42

the cable is very short, 25cm plus or minus and yes there is a PT100 daughter board installed.
You suggest to use the micro SD instead? I have some doubts with the micro SD Card due to their structure....

gate546 · 14 Jun 2018, 17:47

@gate546 said in DAMNED CRASH - Stuck in spin loop:

@dc42 said in DAMNED CRASH - Stuck in spin loop:

@gate546 said in DAMNED CRASH - Stuck in spin loop:

One more info, the involved gcode file is over a normal SD card that is installed on the 7" Panel and NOT on the micro SD Card that is inside the Duet Wifi board

That may be very relevant. The SPI link to the SD card on the PanelDue doesn't have the same hardware timeout mechanisms as the interface to the micro SD card. Also if the ribbon cable is too long then the signal integrity may be degraded. For best reliability, I recommend printing from the micro SD card.

Do you have any other SPI-connected devices, for example a PT100 or thermocouple daughter board?

@dc42

the cable is very short, 25cm plus or minus and yes there is a PT100 daughter board installed.
You suggest to use the micro SD instead? I have some doubts with the micro SD Card due to their structure....
@dc42
David, just for information, to close the circle.
The same long print that caused the crash I wrote before, ended happily without any kind of problems
Same machine, same SD card, all the same, the difference is that I never tried to connect to printer with the browser.
Perhaps we should investigate very well in this field.