Magnetic Sensor - Extruder X reports sensor not working errors

clegg78

I've ended up disabling the sensors on my system for now, and may remove them entirely until a solution can be found that lets them work consistently. As someone else has posted (and from my searching "Not Working" errors seem to be tied to people running the DUEX boards as well. ) I cant test if they don't have the issues without the Duex as it's needed to run my printer.

IMO this is a Duet firmware/hardware issue that needs to be resolved. I even changed the wires over to shielded/grounded wires to ensure there was no interference in the loop.

At this point the sensors are a liability and I cant trust them.

clegg78

@droftarts

So I was doing some testing after having both sensors disabled for the last few weeks. Once I enable them I start seeing Error codes and output buffer overruns. If I disable them, I never see error codes. Regardless if they are on the CONN_LCD or on normal end stops.

droftarts

@clegg78 There was a recent very lengthy thread regarding someone who had problems with errors and underruns, and it came down to having wires running next to or along the ribbon cable that connects the Duet and Duex. This is because the ribbon cable can pick up interference from other cables relatively easily. Can you check your cables don't run near the ribbon cable?

Ian

clegg78

@droftarts

I will check that. Currently the only wires running near the ribbon for the Duex are power 24v runs, everything else is either sheilded or away from the ribbon wire. Would a shorter wire help limit that issue? My Duet and Duex are very close to one another, and I could shorten that wire considerably. I will work on rerouting some wiring away from that connector. The Conn_LCD connector is close to the ribbon in general but the wires run away from the connector for the sensors.

I will do some testing and checking on that. I'll also look for that thread!

Thanks

droftarts

@clegg78 said in Magnetic Sensor - Extruder X reports sensor not working errors:

I'll also look for that thread!

Started in this one: https://forum.duet3d.com/topic/15421/duet-2-05-memory-leak
Continued here, with final identification of problem here: https://forum.duet3d.com/post/149785

Ian

clegg78

@droftarts Thanks!

I just finished upgrading to RRF3 3.01 RC12 just to keep current, I am going to shorten the ribbon for the Duex and wrap it in aluminized tape, and move some of the wires away from it. I will report back!

clegg78

@droftarts

So I shortened the ribbon for the Duet/Duex by about 1.5" (I know how to re-terminate make 50 pin cables from my SCSI days) I moved all the wires away from the ribbon as well.

I powered up the printer running RRF 3.01RC12, and went to lunch to work on it later. All I did was enable the 2 sensors, and my standard config (updated for RRF3).

When I came back from lunch I ran M122 and saw it ran into a buffer exhaustion issues while it was just sitting here!!

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.01-RC12 running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DGM-917NK-F23T0-6J1F6-3SD6T-1GBWD
Used output buffers: 3 of 24 (24 max)
=== RTOS ===
Static ram: 28084
Dynamic ram: 97540 of which 44 recycled
Exception stack ram used: 536
Never used ram: 4868
Tasks: NETWORK(ready,176) SENSORS(blocked,112) HEAT(blocked,1228) DUEX(suspended,160) MAIN(running,1828) IDLE(ready,76)
Owned mutexes:
=== Platform ===
Last reset 00:43:24 ago, cause: power up
Last software reset at 2020-05-14 10:59, reason: User, spinning module GCodes, available RAM 4936 bytes (slot 2)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 4
MCU temperature: min 41.1, current 41.7, max 42.2
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 2020-05-14 12:08:16
Cache data hit count 4294967295
Slowest loop: 12.49ms; fastest: 0.15ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.5ms, write time 4.4ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 12, completed moves: 12, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 11.49ms; fastest: 0.05ms
Responder states: HTTP(2) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
Interface state active, link 100Mbps full duplex
=== Filament sensors ===
Extruder 0: pos 0.00, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
Extruder 1: pos 0.00, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
=== DueX ===
Read count 0, 0.00 reads/min

The wire closest to the ribbon there is an LED that turns on only when the printer is paused.

droftarts

@clegg78 well, it was worth a shot, I suppose. I’ll escalate this for @dc42 to look at.

Ian

clegg78

@droftarts @dc42 Thanks!

So tonight I was doing a long 7 hour print, after upgrading to RRF 3.1 on my Duet 2 Ethernet.

The Sensor on E0 that was printing worked fine for ~ the first 4 or 5 hours. Then it had the first "Sensor not working" error, then a second about 4hours later.

The weird thing was after the last one, the sensor went dark. No LED flashing, nothing... After the print I plugged it back in and it started flashing again. Could there be some static discharge issue at play here? (correction - the last event was a "too little movement error" when the sensor was visibly dead) The E0 sensor is the older of the two, I RMA'd the E1 sensor already.

I did see an Error Code 4 come up in the M122, but that happened about hour 2 or 3 into the print, before any "Not Working" error happened.

I am onto another print that I need to get done tonight, but here is the M122 from that (Note I disabled the E0 sensor for this print as I cant have it stop overnight)

m122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.0 running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DGM-917NK-F23T0-6J1F6-3SD6T-1GBWD
Used output buffers: 3 of 24 (24 max)
=== RTOS ===
Static ram: 28180
Dynamic ram: 97052 of which 464 recycled
Exception stack ram used: 600
Never used ram: 4776
Tasks: NETWORK(ready,36) SENSORS(blocked,112) HEAT(blocked,1224) DUEX(suspended,160) MAIN(running,1732) IDLE(ready,80)
Owned mutexes:
=== Platform ===
Last reset 12:10:33 ago, cause: software
Last software reset at 2020-05-19 10:25, reason: Stack overflow, spinning module none, available RAM 4932 bytes (slot 2)
Software reset code 0x4111 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80e BFAR 0xe000ed38 SP 0x2001ffb4 Task NETW
Stack: 20002860 20002894 00455625 00000000 00000000 200029b8 20002a55 0000000a 00454c35 200028c4 200055f4 
Error status: 4
MCU temperature: min 42.3, current 43.4, max 43.8
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: ok, SG min/max 25/1023
Driver 1: standstill, SG min/max 170/1023
Driver 2: standstill, SG min/max 161/1023
Driver 3: ok, SG min/max 0/1023
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max 337/1023
Driver 6: ok, SG min/max 18/1023
Driver 7: ok, SG min/max 18/1023
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 2020-05-19 22:36:08
Cache data hit count 4294967295
Slowest loop: 8.91ms; fastest: 0.14ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 8
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 5.2ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 162, MinFreeDm: 143, MaxWait: 976200ms
Bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 5667, completed moves: 5647, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: 3
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.1
Heater 1 is on, I-accum = 0.6
=== GCodes ===
Segments left: 1
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X-46.276 Y21.579 F3705" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 17.16ms; fastest: 0.05ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
Interface state active, link 100Mbps full duplex
=== Filament sensors ===
Extruder 1: pos 0.00, errs: frame 0 parity 0 ovrun 0 pol 1 ovdue 0
=== DueX ===
Read count 0, 0.00 reads/min

The Event log:

2020-05-19 15:07:42 Started printing file 0:/gcodes/K40ElbowMK1.gcode
2020-05-19 15:37:59 Resume state saved
2020-05-19 15:38:02 Printing paused at X-16.8 Y-53.9 Z1.2 U201.0
2020-05-19 15:41:02 Printing resumed
2020-05-19 15:41:23 Resume state saved
2020-05-19 15:41:26 Printing paused at X-32.5 Y-63.7 Z1.2 U201.0
2020-05-19 15:41:39 Printing resumed
2020-05-19 15:55:19 HTTP client 192.168.50.122 login succeeded
2020-05-19 17:32:47 HTTP client 192.168.50.122 disconnected
2020-05-19 17:32:58 HTTP client 192.168.50.122 disconnected
2020-05-19 17:33:00 HTTP client 192.168.50.122 login succeeded
2020-05-19 17:33:51 HTTP client 192.168.50.122 disconnected
2020-05-19 19:23:41 Resume state saved
2020-05-19 19:23:41 Extruder 0 reports sensor not working
2020-05-19 19:23:45 Printing paused at X47.1 Y8.1 Z50.1 U201.0
2020-05-19 19:24:10 HTTP client 192.168.50.137 login succeeded
2020-05-19 19:24:40 Printing resumed
2020-05-19 21:14:35 HTTP client 192.168.50.137 login succeeded
2020-05-19 21:16:48 HTTP client 192.168.50.137 login succeeded
2020-05-19 21:16:57 HTTP client 192.168.50.137 login succeeded
2020-05-19 22:03:39 Resume state saved
2020-05-19 22:03:39 Extruder 0 reports too little movement
2020-05-19 22:03:42 Printing paused at X-3.8 Y-54.5 Z110.1 U201.0
2020-05-19 22:04:47 HTTP client 192.168.50.137 login succeeded
2020-05-19 22:04:48 HTTP client 192.168.50.137 login succeeded
2020-05-19 22:05:18 Printing resumed
2020-05-19 22:11:53 Finished printing file 0:/gcodes/K40ElbowMK1.gcode, print time was 6h 58m
2020-05-19 22:24:44 Started printing file 0:/gcodes/Elbow Top.gcode

droftarts

@clegg78 Can you post the response to M591 D0 and M591 D1? Though I think you'll need to run both of them for a short while to get any data. Are they still connected to Z and E1 endstops?

It still feels like an SPI problem between the Duet and Duex to me, ie something glitchy in communication between the two causing the error 4 (Output buffer starvation). I don't know if it is the sensors causing it, or the sensors misreading as a result. One thing I notice is from your config.g is that you have the X axis (X and U) split, with X on the Duet and U on the Duex. I guess you combine these to X with M584 X0:5 after homing X? My thinking is that it would be better to have the motors of each axis on the same driver, and have X, U and Y on the Duet, Z and extruders on the Duex, if possible.

Ian

clegg78

@droftarts said in Magnetic Sensor - Extruder X reports sensor not working errors:

till feels like an SPI problem between the Duet and Duex to me, ie something glitchy in communication between the two causing the error 4 (Output buffer starvation). I don't know if it is the sensors causing it, or the sensors misreading as a result. One thing I notice is from your config.g is that you have the X axis (X and U) split, with X on the Duet and U on the Duex. I guess you combine these to X with M584 X0:5 after homing X? My thinking is that it would be better to have the motors of each axis on the same driver, and have X, U and Y on the Duet, Z and extruders on the Duex, if possible.

Yeah I am about to print again and I will get the info from them for you.

Also I noticed something interesitng, I just connected to the printer after it was sitting idle for a few hours after powering up. and on connection I heard it reboot the machine. I checked the M122 and saw this:

Last reset 00:00:23 ago, cause: software
Last software reset at 2020-05-20 13:46, reason: Stack overflow, spinning module none, available RAM 4932 bytes (slot 3)
Software reset code 0x4111 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80e BFAR 0xe000ed38 SP 0x2001ffb4 Task NETW
Stack: 20002860 20002894 00455625 00000000 00000000 200029b8 00000000 a0000000 00454c35 200028c4 200055f4 
Error status: 0

A Stack Overflow reset out of nowhere? Not sure I've seen that before.

dc42

@clegg78 said in Magnetic Sensor - Extruder X reports sensor not working errors:

A Stack Overflow reset out of nowhere? Not sure I've seen that before.

That's a known issue with RRF 3.1.0, fixed in 3.1.1.

dc42

Have all your "sensor not working" errors been accompanied by the sensor no longer flashing?

There could indeed be a static discharge issue. Have you grounded the body of the extruder stepper motor?

clegg78

@dc42

No that was the first time I've seen that happen, after unplugging and replugging it, it worked again. The Extruder body isnt grounded (its on the X Axis gantry, direct drive extruder, so no easy way to ground it. I dont have an extra wire in the bundle going to the printer right now that carries a ground consistently. I could look into running a small ground wire in the bundle to the extruders.

And yeah upgrading to 3.1.1 fixed that stack overflow error.

Thalios

I've been running 3.1.1 for a bit and just tonight, ran in a similar problem. I'll create a new thread with the exact issue.