Reset during print Part 2 - Duet 3.1.1 / 3.2beta3 - Duet3+SBC

dc42

@Via said in Reset during print Part 2 - Duet 3.1.1 / 3.2beta3 - Duet3+SBC:

@dc42 Many thanks.

Is it worth trying with DuetPi Lite? Was going to try earlier but work got in the way

I doubt it. The assertion failure looks like it was caused either by memory corruption, or possibly by a power brownout. I guess a hardware problem is also a possibility.

Can you set up a macro that runs the file in simulation mode within a loop, to see if you can provoke the failure that way?

Via

@dc42 I've ran the file as a normal simulation which completes fine.

Trying to do a macro to loop it but my gcode knowledge is terrible and keep freezing the board can you point me in right direction?

Was thinking this (I know never ending loops are bad in themselves but was just going to let it run until I reset it) but looks like it is just buffering all the M24s in one go how would I go about waiting for it to finish one before starting the next?

M37 S1
M23 "[PLA] CFFFP_Top - ReRender - zero temps.gcode"
while true
	M24

Via

@dc42 Stopped overthinking it and just added the M37/M23/M24 to the end of gcode which is looping properly now.

Has done ~130 loops with no issues in simulation.

Via

Simulation loop ran for 4 1/2 hours (~380 loops) with no issues.

Ripped 24V PSU out of another printer today and unplugged heaters and fans so just steppers and sensors plugged in and done another test print, again this did not complete.

This time M122 back to Memory Protection Fault

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2-beta3.2 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD
Used output buffers: 2 of 40 (11 max)
=== RTOS ===
Static ram: 122236
Dynamic ram: 168580 of which 376 recycled
Never used RAM 101000, free system stack 200 words
Tasks: NETWORK(ready,161) ETHERNET(blocked,109) HEAT(blocked,297) CanReceiv(blocked,948) CanSender(blocked,371) CanClock(blocked,356) TMC(blocked,54) MAIN(running,1111) IDLE(ready,19)
Owned mutexes:
=== Platform ===
Last reset 00:00:20 ago, cause: software
Last software reset at 2020-11-21 19:46, reason: MemoryProtectionFault mmarValid daccViol, Platform spinning, available RAM 101000, slot 1
Software reset code 0x4160 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x00427804 BFAR 0x00038100 SP 0x2040aa78 Task TMC
Stack: 00000001 2040a928 00000000 00410210 00000000 00469d0f 00415682 21000000 427a0001 3c800001 00000000 3d000001 00000000 00000000 37533333 43d697a0 00000000 41dde946 43fa0000 44a80000 2040ff38 00469d0f 00000000 40078000 20400020 00415681 0000001e
Error status: 0x00
MCU temperature: min 39.2, current 39.9, max 40.0
Supply voltage: min 23.9, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0
Driver 0: position 0, standstill, reads 35967, writes 14 timeouts 0, SG min/max 0/0
Driver 1: position 0, standstill, reads 35968, writes 14 timeouts 0, SG min/max 0/0
Driver 2: position 0, standstill, reads 35969, writes 14 timeouts 0, SG min/max 0/0
Driver 3: position 0, standstill, reads 35970, writes 14 timeouts 0, SG min/max 0/0
Driver 4: position 0, standstill, reads 35971, writes 14 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 35975, writes 11 timeouts 0, SG min/max 0/0
Date/time: 2020-11-21 19:46:54
Slowest loop: 6.26ms; fastest: 0.15ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 0.8ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 5.46ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== CAN ===
Messages sent 82, send timeouts 82, longest wait 0ms for type 0, free CAN buffers 46

dc42

@Via thanks, I will look into that data later.

dc42

@Via, please post the config.g file that you were using with 3.2beta3.2 when that assertion failure occurred.

Do you have any filament monitors configured? [Edit: from your M122 report i think that you do not.]

Via

@dc42 Thank you.

Config attached, not been changed from assertation failure (or from 3.1.1). No filament monitors.

config.g

Via

@dc42 any update on this?

dc42

I wasn't able to pin this down, except to say that it looks like random memory failure or corruption. Please upgrade to 3.4beta4. If it happens again, please post the print file and config.g if you haven't already (I'm using a tablet, so I can't tell), also daemon.g if you are using it.

Via

@dc42 Failed again on beta4 (M122 at bottom) they are posted already but will group together here.

config.g

[PLA] CFFFP_Top - ReRender - zero temps (1).gcode

full sys directory: https://www.viaraix.net/duet3-sys.zip

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2-beta4 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD
Used output buffers: 1 of 40 (11 max)
=== RTOS ===
Static ram: 123212
Dynamic ram: 168584 of which 704 recycled
Never used RAM 99692, free system stack 188 words
Tasks: NETWORK(ready,169) ETHERNET(blocked,110) HEAT(blocked,298) CanReceiv(blocked,947) CanSender(blocked,371) CanClock(blocked,358) TMC(blocked,54) MAIN(running,1113) IDLE(ready,19)
Owned mutexes:
=== Platform ===
Last reset 00:00:19 ago, cause: software
Last software reset at 2020-11-27 11:41, reason: MemoryProtectionFault mmarValid daccViol, GCodes spinning, available RAM 99692, slot 2
Software reset code 0x4163 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x0044a804 BFAR 0x00000000 SP 0x204100c8 Task MAIN
Stack: 00000002 ffffffff 00434a29 20410308 000002bc 00469483 0046a3c8 610f0000 2042565c 004691bf 2042565c 00000000 000002bb 0000000a 2042565c 204256c8 00000000 00000001 00000000 00468e53 204256c8 00000000 204256c8 004691bf 00000000 00000000 20425540
Error status: 0x00
MCU temperature: min 39.0, current 39.4, max 39.9
Supply voltage: min 23.9, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0
Driver 0: position 0, standstill, reads 32774, writes 14 timeouts 0, SG min/max 0/0
Driver 1: position 0, standstill, reads 32775, writes 14 timeouts 0, SG min/max 0/0
Driver 2: position 0, standstill, reads 32776, writes 14 timeouts 0, SG min/max 0/0
Driver 3: position 0, standstill, reads 32777, writes 14 timeouts 0, SG min/max 0/0
Driver 4: position 0, standstill, reads 32779, writes 14 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 32782, writes 11 timeouts 0, SG min/max 0/0
Date/time: 2020-11-27 11:41:21
Slowest loop: 6.06ms; fastest: 0.15ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 0.7ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 375, MinFreeDm: 375, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 17.56ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== CAN ===
Messages queued 79, send timeouts 177, received 0, lost 0, longest wait 0ms for reply type 0, free buffers 47

dc42

Thanks for your new report. I am fairly certain that the memory protection fault was caused by a hardware error, because there is no way that the instruction at that address should fault, nor any way that the faulting address should be zero. So I suggest we swap your board. Who did you purchase it from?

Via

@dc42 thanks for your time looking into this. Purchase was from E3D but I have just gone over the 6 months now (purchased 13th May), will you be able to authorise warranty replacement as reported prior?

dc42

Yes I approve a warranty replacement.

Via

@dc42 Thank you, will get in touch with E3D. Many thanks for your help.

dc42

@Via, two more things to try:

Try the new binary at https://www.dropbox.com/sh/tehhdvunh0pgr7q/AADgJ4Qj7W18MYaRRNNDdhpwa?dl=0, although I think it likely that it won't make any difference.
Your MCU temperature may be a little high, depending on how accurate the 39.4C reading is. 40C is well within limits, but the actual temperature could be higher. Please try powering up the board when it is cold and immediately run M122 to check the temperature reading. That will allow you to calibrate the MCU temperature sensor. See https://duet3d.dozuki.com/Wiki/Calibrating_the_CPU_temperature.

Via

@dc42 MCU was over reading by ~2C so adjusted to compensate and showing ~35C now.

Installed new firmware and...

30/11/2020, 13:26:02	Finished printing file 0:/gcodes/[PLA] CFFFP_Top - ReRender - zero temps.gcode, print time was 1h 41m

So looked promising so without changing anything I reran print to make sure it was not a fluke and sure enough it was ...

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2-beta4+4 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6J9FA-3S86T-1B5LD
Used output buffers: 1 of 40 (12 max)
=== RTOS ===
Static ram: 123212
Dynamic ram: 168584 of which 704 recycled
Never used RAM 99692, free system stack 122 words
Tasks: NETWORK(ready,165) ETHERNET(blocked,105) HEAT(blocked,297) CanReceiv(blocked,947) CanSender(blocked,352) CanClock(blocked,349) TMC(blocked,20) MAIN(running,1113) IDLE(ready,19)
Owned mutexes:
=== Platform ===
Last reset 00:12:51 ago, cause: software
Last software reset at 2020-11-30 14:18, reason: MemoryProtectionFault mmarValid daccViol, Move spinning, available RAM 99692, slot 1
Software reset code 0x4164 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x00427004 BFAR 0xb6d799a8 SP 0x2045ff58 Task MAIN
Stack: 00000060 000000c0 00000091 96969694 00000001 00468c29 00469614 8100000e 00455a21 0045f237 00000000 2040e064 a5000001 008e0d8e 2040e064 a5000001 00000000 00000001 204281fc 00469d17 0000a5c7 204103b4 2042820c 204281f0 204281fc 20428370 00000000
Error status: 0x00
MCU temperature: min 34.2, current 34.3, max 36.3
Supply voltage: min 23.9, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0
Driver 0: position -3200, standstill, reads 61020, writes 17 timeouts 0, SG min/max 0/156
Driver 1: position -800, standstill, reads 61021, writes 17 timeouts 0, SG min/max 0/319
Driver 2: position 2000, standstill, reads 61022, writes 17 timeouts 0, SG min/max 0/184
Driver 3: position 0, standstill, reads 61024, writes 17 timeouts 0, SG min/max 0/182
Driver 4: position 0, standstill, reads 61027, writes 14 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 61031, writes 11 timeouts 0, SG min/max 0/0
Date/time: 2020-11-30 14:31:40
Slowest loop: 12.11ms; fastest: 0.21ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 0.9ms, write time 0.0ms, max retries 0
=== Move ===
FreeDm 375 (min 372), maxWait 727688ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 13, completed moves 13, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 1], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 12.92ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== CAN ===
Messages queued 3089, send timeouts 6944, received 0, lost 0, longest wait 0ms for reply type 0, free buffers 47

dc42

Thanks, please go ahead with the warranty exchange.

Via

@dc42 received replacement board from E3D today and working as expected in standalone, will get round to putting SBC back in this week but happy there shouldn't be any further issues.

Thank's again to yourself and @chrishamm for the help

jrockland

@dc42 is there a way to properly "flush" the memory on thoses boards ?
I have been working on that issue for couples of weeks now on different setups and boards configuration and it seem to happen more and more often on most of them. I wonder if it could be caused by a bad sector or a bug and because of that spot it just happen more and more. (Sorry for my english here, Im french .. trying to explain clairly..)

Everytime I update/flash the firmware it seem to crash faster too.

At the moment Im down to testing different sd cards, maybe I got some fakes/cheaper cards causing the issue too.

Also important point: the setups I have who never been updated (some benchmarks and extruders tester) dont have that issue. Yet..