Magnetic Sensor - Extruder X reports sensor not working errors

droftarts

@clegg78 said in Magnetic Sensor - Extruder X reports sensor not working errors:

I'll also look for that thread!

Started in this one: https://forum.duet3d.com/topic/15421/duet-2-05-memory-leak
Continued here, with final identification of problem here: https://forum.duet3d.com/post/149785

Ian

clegg78

@droftarts Thanks!

I just finished upgrading to RRF3 3.01 RC12 just to keep current, I am going to shorten the ribbon for the Duex and wrap it in aluminized tape, and move some of the wires away from it. I will report back!

clegg78

@droftarts

So I shortened the ribbon for the Duet/Duex by about 1.5" (I know how to re-terminate make 50 pin cables from my SCSI days) I moved all the wires away from the ribbon as well.

I powered up the printer running RRF 3.01RC12, and went to lunch to work on it later. All I did was enable the 2 sensors, and my standard config (updated for RRF3).

When I came back from lunch I ran M122 and saw it ran into a buffer exhaustion issues while it was just sitting here!!

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.01-RC12 running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DGM-917NK-F23T0-6J1F6-3SD6T-1GBWD
Used output buffers: 3 of 24 (24 max)
=== RTOS ===
Static ram: 28084
Dynamic ram: 97540 of which 44 recycled
Exception stack ram used: 536
Never used ram: 4868
Tasks: NETWORK(ready,176) SENSORS(blocked,112) HEAT(blocked,1228) DUEX(suspended,160) MAIN(running,1828) IDLE(ready,76)
Owned mutexes:
=== Platform ===
Last reset 00:43:24 ago, cause: power up
Last software reset at 2020-05-14 10:59, reason: User, spinning module GCodes, available RAM 4936 bytes (slot 2)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 4
MCU temperature: min 41.1, current 41.7, max 42.2
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 2020-05-14 12:08:16
Cache data hit count 4294967295
Slowest loop: 12.49ms; fastest: 0.15ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.5ms, write time 4.4ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 12, completed moves: 12, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 11.49ms; fastest: 0.05ms
Responder states: HTTP(2) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
Interface state active, link 100Mbps full duplex
=== Filament sensors ===
Extruder 0: pos 0.00, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
Extruder 1: pos 0.00, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
=== DueX ===
Read count 0, 0.00 reads/min

The wire closest to the ribbon there is an LED that turns on only when the printer is paused.

droftarts

@clegg78 well, it was worth a shot, I suppose. I’ll escalate this for @dc42 to look at.

Ian

clegg78

@droftarts @dc42 Thanks!

So tonight I was doing a long 7 hour print, after upgrading to RRF 3.1 on my Duet 2 Ethernet.

The Sensor on E0 that was printing worked fine for ~ the first 4 or 5 hours. Then it had the first "Sensor not working" error, then a second about 4hours later.

The weird thing was after the last one, the sensor went dark. No LED flashing, nothing... After the print I plugged it back in and it started flashing again. Could there be some static discharge issue at play here? (correction - the last event was a "too little movement error" when the sensor was visibly dead) The E0 sensor is the older of the two, I RMA'd the E1 sensor already.

I did see an Error Code 4 come up in the M122, but that happened about hour 2 or 3 into the print, before any "Not Working" error happened.

I am onto another print that I need to get done tonight, but here is the M122 from that (Note I disabled the E0 sensor for this print as I cant have it stop overnight)

m122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.0 running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DGM-917NK-F23T0-6J1F6-3SD6T-1GBWD
Used output buffers: 3 of 24 (24 max)
=== RTOS ===
Static ram: 28180
Dynamic ram: 97052 of which 464 recycled
Exception stack ram used: 600
Never used ram: 4776
Tasks: NETWORK(ready,36) SENSORS(blocked,112) HEAT(blocked,1224) DUEX(suspended,160) MAIN(running,1732) IDLE(ready,80)
Owned mutexes:
=== Platform ===
Last reset 12:10:33 ago, cause: software
Last software reset at 2020-05-19 10:25, reason: Stack overflow, spinning module none, available RAM 4932 bytes (slot 2)
Software reset code 0x4111 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80e BFAR 0xe000ed38 SP 0x2001ffb4 Task NETW
Stack: 20002860 20002894 00455625 00000000 00000000 200029b8 20002a55 0000000a 00454c35 200028c4 200055f4 
Error status: 4
MCU temperature: min 42.3, current 43.4, max 43.8
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: ok, SG min/max 25/1023
Driver 1: standstill, SG min/max 170/1023
Driver 2: standstill, SG min/max 161/1023
Driver 3: ok, SG min/max 0/1023
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max 337/1023
Driver 6: ok, SG min/max 18/1023
Driver 7: ok, SG min/max 18/1023
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 2020-05-19 22:36:08
Cache data hit count 4294967295
Slowest loop: 8.91ms; fastest: 0.14ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 8
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 5.2ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 162, MinFreeDm: 143, MaxWait: 976200ms
Bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 5667, completed moves: 5647, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: 3
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.1
Heater 1 is on, I-accum = 0.6
=== GCodes ===
Segments left: 1
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X-46.276 Y21.579 F3705" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 17.16ms; fastest: 0.05ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
Interface state active, link 100Mbps full duplex
=== Filament sensors ===
Extruder 1: pos 0.00, errs: frame 0 parity 0 ovrun 0 pol 1 ovdue 0
=== DueX ===
Read count 0, 0.00 reads/min

The Event log:

2020-05-19 15:07:42 Started printing file 0:/gcodes/K40ElbowMK1.gcode
2020-05-19 15:37:59 Resume state saved
2020-05-19 15:38:02 Printing paused at X-16.8 Y-53.9 Z1.2 U201.0
2020-05-19 15:41:02 Printing resumed
2020-05-19 15:41:23 Resume state saved
2020-05-19 15:41:26 Printing paused at X-32.5 Y-63.7 Z1.2 U201.0
2020-05-19 15:41:39 Printing resumed
2020-05-19 15:55:19 HTTP client 192.168.50.122 login succeeded
2020-05-19 17:32:47 HTTP client 192.168.50.122 disconnected
2020-05-19 17:32:58 HTTP client 192.168.50.122 disconnected
2020-05-19 17:33:00 HTTP client 192.168.50.122 login succeeded
2020-05-19 17:33:51 HTTP client 192.168.50.122 disconnected
2020-05-19 19:23:41 Resume state saved
2020-05-19 19:23:41 Extruder 0 reports sensor not working
2020-05-19 19:23:45 Printing paused at X47.1 Y8.1 Z50.1 U201.0
2020-05-19 19:24:10 HTTP client 192.168.50.137 login succeeded
2020-05-19 19:24:40 Printing resumed
2020-05-19 21:14:35 HTTP client 192.168.50.137 login succeeded
2020-05-19 21:16:48 HTTP client 192.168.50.137 login succeeded
2020-05-19 21:16:57 HTTP client 192.168.50.137 login succeeded
2020-05-19 22:03:39 Resume state saved
2020-05-19 22:03:39 Extruder 0 reports too little movement
2020-05-19 22:03:42 Printing paused at X-3.8 Y-54.5 Z110.1 U201.0
2020-05-19 22:04:47 HTTP client 192.168.50.137 login succeeded
2020-05-19 22:04:48 HTTP client 192.168.50.137 login succeeded
2020-05-19 22:05:18 Printing resumed
2020-05-19 22:11:53 Finished printing file 0:/gcodes/K40ElbowMK1.gcode, print time was 6h 58m
2020-05-19 22:24:44 Started printing file 0:/gcodes/Elbow Top.gcode

droftarts

@clegg78 Can you post the response to M591 D0 and M591 D1? Though I think you'll need to run both of them for a short while to get any data. Are they still connected to Z and E1 endstops?

It still feels like an SPI problem between the Duet and Duex to me, ie something glitchy in communication between the two causing the error 4 (Output buffer starvation). I don't know if it is the sensors causing it, or the sensors misreading as a result. One thing I notice is from your config.g is that you have the X axis (X and U) split, with X on the Duet and U on the Duex. I guess you combine these to X with M584 X0:5 after homing X? My thinking is that it would be better to have the motors of each axis on the same driver, and have X, U and Y on the Duet, Z and extruders on the Duex, if possible.

Ian

clegg78

@droftarts said in Magnetic Sensor - Extruder X reports sensor not working errors:

till feels like an SPI problem between the Duet and Duex to me, ie something glitchy in communication between the two causing the error 4 (Output buffer starvation). I don't know if it is the sensors causing it, or the sensors misreading as a result. One thing I notice is from your config.g is that you have the X axis (X and U) split, with X on the Duet and U on the Duex. I guess you combine these to X with M584 X0:5 after homing X? My thinking is that it would be better to have the motors of each axis on the same driver, and have X, U and Y on the Duet, Z and extruders on the Duex, if possible.

Yeah I am about to print again and I will get the info from them for you.

Also I noticed something interesitng, I just connected to the printer after it was sitting idle for a few hours after powering up. and on connection I heard it reboot the machine. I checked the M122 and saw this:

Last reset 00:00:23 ago, cause: software
Last software reset at 2020-05-20 13:46, reason: Stack overflow, spinning module none, available RAM 4932 bytes (slot 3)
Software reset code 0x4111 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80e BFAR 0xe000ed38 SP 0x2001ffb4 Task NETW
Stack: 20002860 20002894 00455625 00000000 00000000 200029b8 00000000 a0000000 00454c35 200028c4 200055f4 
Error status: 0

A Stack Overflow reset out of nowhere? Not sure I've seen that before.

dc42

@clegg78 said in Magnetic Sensor - Extruder X reports sensor not working errors:

A Stack Overflow reset out of nowhere? Not sure I've seen that before.

That's a known issue with RRF 3.1.0, fixed in 3.1.1.

dc42

Have all your "sensor not working" errors been accompanied by the sensor no longer flashing?

There could indeed be a static discharge issue. Have you grounded the body of the extruder stepper motor?

clegg78

@dc42

No that was the first time I've seen that happen, after unplugging and replugging it, it worked again. The Extruder body isnt grounded (its on the X Axis gantry, direct drive extruder, so no easy way to ground it. I dont have an extra wire in the bundle going to the printer right now that carries a ground consistently. I could look into running a small ground wire in the bundle to the extruders.

And yeah upgrading to 3.1.1 fixed that stack overflow error.

Thalios

I've been running 3.1.1 for a bit and just tonight, ran in a similar problem. I'll create a new thread with the exact issue.