Filament error on extruder 0: sensorError

Aurimas

Duet magnetic filament sensor
connected to toolboard.
runs sometimes for days non stop in a single print and then crashes out

@dc42

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.3 (2024-01-24 17:56:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: AGMAV-AR6KL-K65J0-409N6-LGW1Z-7XF1B
Used output buffers: 22 of 40 (40 max)
=== RTOS ===
Static ram: 103200
Dynamic ram: 123876 of which 348 recycled
Never used RAM 10104, free system stack 102 words
Tasks: NETWORK(2,nWait 7,141.1%,174) HEAT(3,nWait 6,3.2%,326) Move(4,nWait 6,126.0%,239) CanReceiv(6,nWait 1,27.6%,563) CanSender(5,nWait 7,8.8%,328) CanClock(7,delaying,0.9%,349) TMC(4,nWait 6,178.5%,68) MAIN(1,running,102.6%,663) IDLE(0,ready,134.1%,30) AIN(4,delaying,106.2%,256), total 829.2%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 96:18:58 ago, cause: software
Last software reset at 2024-05-04 00:45, reason: HardFault imprec, Platform spinning, available RAM 10364, slot 0
Software reset code 0x4060 HFSR 0x40000000 CFSR 0x00000400 ICSR 0x0047d803 BFAR 0xe000ed38 SP 0x20012008 Task NETW Freestk 489 ok
Stack: 2002c5e0 200324d0 200014e8 00000000 2003366e 00030019 0002ff30 61016000 200361d0 2002c5c8 200187fc 2002c41e 200187fc 2001e9f8 0003019f 00000000 00000000 00000000 200120a0 00000014 46c577d2 00000002 dbe60050 6a01a8c0 08000162 00000001 00034861
Error status: 0x04
Aux0 errors 0,1,250
MCU revision 3, ADC conversions started 351223473, completed 351223473, timed out 0, errs 0
MCU temperature: min 20.5, current 34.1, max 41.8
Supply voltage: min 23.2, current 23.7, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/1, heap memory allocated/used/recyclable 2048/356/280, gc cycles 0
Events: 8 queued, 8 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 57, reads 64318, writes 25, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 57, reads 64317, writes 25, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x71
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 49, reads 64322, writes 21, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 49, reads 64321, writes 21, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x6f
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 49, reads 64322, writes 21, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 57, reads 64317, writes 25, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x6f
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 57, reads 64317, writes 25, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x71
Date/time: 2024-05-08 07:42:34
Cache data hit count 4294967295
Slowest loop: 239.99ms; fastest: 0.09ms
=== Storage ===
Free file entries: 17
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 19.8ms, write time 37.7ms, max retries 0
=== Move ===
DMs created 83, segments created 43, maxWait 29203796ms, bed compensation in use: mesh, height map offset 0.000, max steps late 1, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 52109, on retry 90684, too short 89620, wrong shape 5062832, maybepossible 40329
=== DDARing 0 ===
Scheduled moves 1586025, completed 1586025, hiccups 0, stepErrors 0, LaErrors 0, Underruns [157, 0, 74], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters 4 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is assembling a command in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x000080f
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
check 0 clear 0
Extruder 0: no data received, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
Extruder 1: no data received, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
=== CAN ===
Messages queued 11853473, received 964391860, lost 0, errs 0, boc 0
Longest wait 4ms for reply type 6029, peak Tx sync delay 551, free buffers 26 (min 24), ts 1733691/1733690/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 213.20ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1.0
MAC address e8:68:e7:e5:b1:d7
Module reset reason: Power up, Vcc 3.37, flash size 2097152, free heap 31620
WiFi IP address 192.168.1.87
Signal strength -38dBm, channel 2, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-rc.3 (2024-01-24 17:55:14)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 1948, free system stack 55 words
Tasks: Move(3,nWait 7,57.8%,75) HEAT(2,nWait 6,35.8%,41) CanAsync(5,nWait 4,0.0%,51) CanRecv(3,nWait 1,20.4%,65) CanClock(5,nWait 1,2.3%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,delaying,41.2%,53) MAIN(1,running,100.0%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,10.4%,112), total 268.0%
Owned mutexes:
Last reset 96:23:26 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 400.0 steps/mm, ok, SG min 0, read errors 177, write errors 1, ifcnt 254, reads 47107, writes 223, timeouts 258, DMA errors 0, CC errors 0, failedOp 0x41, steps req 0 done 1193122278
Moves scheduled 8732507, completed 8732505, in progress 1, hiccups 1531, segs 44, step errors 0, maxLate 1 maxPrep 810, maxOverdue 42562, maxInc 22067, mcErrs 0, gcmErrs 0, ebfmin -1.00 max 1.00
Peak sync jitter -5/18, peak Rx sync delay 270, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 0.2, current 23.7, max 45.5
MCU temperature: min 39.9C, current 75.9C, max 82.2C
Last sensors broadcast 0x00000002 found 1 121 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 958152643, send timeouts 0, received 13246475, lost 0, errs 0, boc 0, free buffers 18, min 17, error reg 0
dup 0, oos 22/0/0/11, bm 0, wbm 0, rxMotionDelay 414, adv 7037/109175
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 3, contentions 0, other errors 0
=== Filament sensors ===
Interrupt 4 to 234us, poll 6 to 4730us
Driver 0: pos 13937.70, errs: frame 748 parity 0 ovrun 0 pol 0 ovdue 0

droftarts

@Aurimas Where and when do you get this error message? Is that the full extent of the message? What does it report if you send M591 D0 (or whichever extruder it is connected to) after that message? Your M122 reports a 'HardFault imprec', which would generally cause the firmware to freeze; did it?

Please note that you are running RRF 3.5.0-rc.3. Is there a reason you haven't updated to the 3.5.1 release? We don't usually support beta and RC releases once the main release is out. If you could confirm the same behaviour in 3.5.1, it may give us something more to go on.

Also, you could update the bootloader on your 1LC board, the current version is 2.8, yours is on 2.4. See https://docs.duet3d.com/en/User_manual/RepRapFirmware/Updating_bootloader

For reference, from @dc42:

'Hardfault invstate' means the MCU tried to execute invalid instructions. This normally means it jumped to a random location. I don't think I've ever seen one of those that was reproducible, or that has a stack trace that suggested a firmware bug; that's why I think ESD or possibly power dropout is a likely cause.

'HardFault imprec' means that because of write buffering, the exact location of the bus fault (which is normally caused by writing to non-existent memory) wasn't recorded.

Also, please see @dc42's note in this thread, which will allow him to track down the issue:

To those of you experiencing this issue, please send the following command after power up, before you start printing:

M122 P500 S0

This will reduce performance a little, however it should change the imprecise Hard Fault resets that I see in your M122 reports to precise ones, which will make this easier for me to diagnose. Then post M122 reports if the problem happens again.

Ian

Aurimas

@droftarts said in Filament error on extruder 0: sensorError:

M122 P500 S0

Thanks Ian,

those M122 outputs are after the error.
and that is all it says.
After the error occurs I resume the print and then it goes for a while until it throws tantrums again

droftarts

@Aurimas Sorry, I think I was focussing on the wrong thing. I see that the Hardfault was the last software reset, 96 hours before the M122 was sent! I think the part to focus on is this:

=== Filament sensors ===
check 0 clear 0
Extruder 0: no data received, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
Extruder 1: no data received, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0

and from M122 B121:

=== Filament sensors ===
Interrupt 4 to 234us, poll 6 to 4730us
Driver 0: pos 13937.70, errs: frame 748 parity 0 ovrun 0 pol 0 ovdue 0

I'm not sure how to read this, as it looks like there are 3 filament sensors configured. Maybe @dc42 can explain what a 'sensorError' means. I would imagine a failure to read the sensor, but why this would occur, I can only speculate. Possibly it's related to the Hardfault, and an ESD event is causing the filament monitor to fail to send data momentarily? Is the hot end metal work and stepper driver grounded through the toolboard?

Please post your filament sensor section of config.g. If possible, if the sensorError reoccurs, send M591 D#, where # is the number of the extruder drive the filament sensor is connected to, and see what it reports. Also check for any loose or broken wiring or poorly crimped connections between the filament sensor and toolboard.

Ian

droftarts

@Aurimas From @dc42 :

The 748 framing errors reported by the tool board is rather high. It might possibly mean that the filament sensor cable is running right next to a stepper motor cable and picking up noise.

He also noticed that you have a high hiccup count on the 1LC:

Driver 0: pos 0, 400.0 steps/mm, ok, SG min 0, read errors 177, write errors 1, ifcnt 254, reads 47107, writes 223, timeouts 258, DMA errors 0, CC errors 0, failedOp 0x41, steps req 0 done 1193122278
Moves scheduled 8732507, completed 8732505, in progress 1, hiccups 1531, segs 44, step errors 0, maxLate 1 maxPrep 810, maxOverdue 42562, maxInc 22067, mcErrs 0, gcmErrs 0, ebfmin -1.00 max 1.00
Peak sync jitter -5/18, peak Rx sync delay 270, resyncs 0/0, no timer interrupt scheduled

One thing I'm not certain of is whether high hiccup counts on a 1LC might indirectly cause errors receiving data from the MFM.

The high hiccup count may be caused by too many steps being asked for per second, ie running the extruder motor too fast.

Ian

Aurimas

@droftarts thanks, but I am not sure how it solves my issue
1LC is mounted in the front of the LGX extruder with filament sensor on top of it.
saying that I had issues with simple switch type filament sensor as well.

Aurimas

@droftarts

straight after the error no data received this time

M591 D0
Duet3D rotating magnet filament monitor v4 on pin 121.io1.in, enabled, sensitivity 24.80mm/rev, allow 50% to 150%, check printing moves every 5.0mm, version 4, mag 132 agc 61, measured sensitivity 25.79mm/rev, min 95% max 105% over 113023.8mm

	M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.4.6 (2023-07-21 14:17:33)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 2108, free system stack 17 words
Tasks: Move(notifyWait,0.0%,99) HEAT(notifyWait,0.4%,75) CanAsync(notifyWait,0.0%,61) CanRecv(notifyWait,0.0%,74) CanClock(notifyWait,0.0%,65) ACCEL(notifyWait,0.0%,61) TMC(notifyWait,3.0%,57) MAIN(running,91.7%,351) IDLE(ready,0.0%,26) AIN(delaying,4.9%,142), total 100.0%
Last reset 07:43:21 ago, cause: software
Last software reset data not available
Driver 0: pos 100779577, 400.0 steps/mm,standstill, SG min 2, read errors 0, write errors 0, ifcnt 55, reads 22960, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 713681, completed 713681, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -1/9, peak Rx sync delay 193, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 23.8, current 24.2, max 24.3
MCU temperature: min 55.0C, current 78.5C, max 81.7C
Last sensors broadcast 0x00000002 found 1 6 ticks ago, 0 ordering errs, loop time 1
CAN messages queued 975, send timeouts 0, received 605, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 0, other errors 0
=== Filament sensors ===
Interrupt 4 to 9us, poll 8 to 1184us
Driver 0: pos 13916.25, errs: frame 236 parity 0 ovrun 0 pol 0 ovdue 0

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.6 (2023-07-21 14:09:13) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: UFKKT-RN6KL-K65J0-409N8-28W1Z-RSKK2
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 103712
Dynamic ram: 112888 of which 304 recycled
Never used RAM 20896, free system stack 112 words
Tasks: NETWORK(notifyWait,2.6%,201) HEAT(notifyWait,0.1%,338) Move(notifyWait,15.2%,235) CanReceiv(notifyWait,0.5%,771) CanSender(notifyWait,0.4%,324) CanClock(delaying,0.0%,349) TMC(notifyWait,7.8%,72) MAIN(running,58.6%,425) IDLE(ready,9.5%,30) AIN(delaying,5.3%,263), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 07:43:06 ago, cause: software
Last software reset details not available
Error status: 0x04
Aux0 errors 0,0,1964
MCU revision 3, ADC conversions started 27789048, completed 27789048, timed out 0, errs 0
Step timer max interval 1487
MCU temperature: min 23.0, current 32.0, max 33.1
Supply voltage: min 21.7, current 23.7, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/1, heap memory allocated/used/recyclable 2048/84/8, gc cycles 0
Events: 6 queued, 6 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 48, reads 31646, writes 2, timeouts 2, DMA errors 0, CC errors 0, failedOp 0x71
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 48, reads 31647, writes 2, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x71
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 40, reads 31649, writes 2, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 40, reads 31649, writes 2, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 40, reads 31648, writes 2, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 0, ifcnt 48, reads 31648, writes 2, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 48, reads 31649, writes 2, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-05-13 16:45:03
Cache data hit count 4294967295
Slowest loop: 229.31ms; fastest: 0.07ms
=== Storage ===
Free file entries: 8
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 11.2ms, write time 4.7ms, max retries 0
=== Move ===
DMs created 83, segments created 44, maxWait 2882877ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 723642, completed 723642, hiccups 0, stepErrors 0, LaErrors 0, Underruns [1, 0, 6], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters 4 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is assembling a command in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== Filament sensors ===
Extruder 0: no data received
Extruder 1: no data received
=== CAN ===
Messages queued 963536, received 1137912, lost 0, boc 0
Longest wait 5ms for reply type 6024, peak Tx sync delay 498, free buffers 18 (min 16), ts 138736/138736/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 203.94ms; fastest: 0.06ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 4 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 6
WiFi firmware version 1.26
WiFi MAC address e8:68:e7:e1:50:c5
WiFi Vcc 3.43, reset reason Power up
WiFi flash size 2097152, free heap 24712
WiFi IP address 192.168.1.87
WiFi signal strength -49dBm, mode 802.11n, reconnections 0, sleep mode modem
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

droftarts

@Aurimas said in Filament error on extruder 0: sensorError:

From the M122 B121 report:

Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)

Bootloader v2.4 is an older version, though there's not much that changes between versions. However, it might be worth seeing if updating this helps. This is easy to do, see https://docs.duet3d.com/en/User_manual/RepRapFirmware/Updating_bootloader
Files are here: https://github.com/Duet3D/Duet3Bootloader/releases/tag/2.8

Never used RAM 2108, free system stack 17 words

I don't think the issue is caused by running out of memory. It doesn't say it has. I'll ask @dc42 if this is particularly low. What is connected to the 1LC?

Driver 0: pos 100779577, 400.0 steps/mm,standstill, SG min 2, read errors 0, write errors 0, ifcnt 55, reads 22960, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 713681, completed 713681, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -1/9, peak Rx sync delay 193, resyncs 0/0, no step interrupt scheduled

Hiccups don't seem to have caused this error, as there are none reported, so I think @dc42's concern about this seems less relevant. What sort of speeds do you usually run at that may have produced the hiccups before? Is it a large nozzle and high layer height, printing very fast?

=== Filament sensors ===
Interrupt 4 to 9us, poll 8 to 1184us
Driver 0: pos 13916.25, errs: frame 236 parity 0 ovrun 0 pol 0 ovdue 0

Again, this shows frame errors, which @dc42 attributes to interference. Could you check wire, crimps and connectors, and possibly refresh the wiring, between the filament sensor and 1LC?

I don't quite understand the Filament sensors section of the M122 report. You only have one filament sensor defined in config.g, and it is connected to the 1LC, correct? Do you have two extruder defined? Could you post the config.g for this machine?

Ian

Aurimas

@droftarts The wires are 100mm long
why would it run sometimes for hours and hours on the same printer and then throw an error every 15min?

droftarts

@Aurimas said in Filament error on extruder 0: sensorError:

why would it run sometimes for hours and hours on the same printer and then throw an error every 15min?

If I knew that, I'd just tell you what to fix!

Ian