Heater Temperature Excursion

Leav · 14 Sept 2020, 21:02

I tried reproducing by extruding material, and was inexplicably able to reproduce the bed heater noise flickering:

@BoA The fact that it's happening on two different thermistors "at the same time" is suspicious to me (i.e. less likely to be a thermistor fault), don't you think?

Phaedrux · 14 Sept 2020, 21:06

How is your wiring for the thermistors ran?

The graph in your first photo is clearly signal noise. It thinks the temp is dropping and increases power to maintain temp, but then it gets a real reading and sees the temp is actually hotter than expected and triggers the fault.

I would be inclined to replace the thermistor and wiring entirely.

Leav · 15 Sept 2020, 19:51

@Phaedrux said in Heater Temperature Excursion:

How is your wiring for the thermistors ran?

The graph in your first photo is clearly signal noise. It thinks the temp is dropping and increases power to maintain temp, but then it gets a real reading and sees the temp is actually hotter than expected and triggers the fault.

I would be inclined to replace the thermistor and wiring entirely.

Thanks for the tip @Phaedrux, in my delta I have 3 wiring harnesses (one in each tower) and the hotend thermistor was running along side the hot end fan for a long distance.

I moved the fan to a different harness and was able to print without errors again, though the noise was significantly higher in the bed thermistor for some reason. (does not go through that harness at all).

I can't really understand why this issue suddenly popped up, and I can't say that I feel like I solved it since the temperature readings for the bed were so noisy in the beginning (but became completely smooth after the bed was up to temp).

Is there perhaps a way to filter transients in the temperature probe? (I meam physically, like a ferrite or a capacitor)

Phaedrux · 15 Sept 2020, 20:00

@Leav said in Heater Temperature Excursion:

Is there perhaps a way to filter transients in the temperature probe? (I meam physically, like a ferrite or a capacitor)

Not sure about that.

Twisted pair and/or shielded cable can help avoid picking up noise. A PT100 in 4 wire mode is a good choice in this situation as it is designed to reject noise.

BoA · 15 Sept 2020, 20:36

@Leav For heatbed this looks like a noise, but honestly... 50*C drop for about 60 seconds? That is not a noise. That looks like poor connection to heater.

Phaedrux · 15 Sept 2020, 20:38

@BoA said in Heater Temperature Excursion:

That looks like poor connection to heater.

If it was actually losing connection it would show open circuit and temp would shoot to 2000c to trigger fault.

BoA · 15 Sept 2020, 20:39

@Phaedrux It would show open circuit in case of problem with thermistor, but on second pic it looks like bad connection to heater.

Leav · 10 Oct 2020, 16:42

Hi,

I'm bringing up this old thread since I am still chasing heater faults...

This happened ~3 hours into a print, after the following changes to the system:

Changed heater cartridge (new)
Changed to thermistor in cartridge form factor
Added silicone sock (even though there is no
Tuned the new heater configuration

Unfortunately I was not near the printer at the time so the temperature graph was lost to the sands of time, but it was and is smooth when I look at it now.

Are there any ideas or suggestions for other things to look for that could cause this? quite annoying to have what looks to be a solid setup interrupted by inexplicable random hiccups like this.

Looking at M122 response, I found this line:

...
=== Platform ===
Slowest loop: 581.48ms; fastest: 0.12ms
...
=== Network ===
Slowest loop: 581.68ms; fastest: 0.00ms
WiFi signal strength -72dBm, reconnections 0, sleep mode modem

That looks like a suspiciously long loop time to me, and could also perhaps explain minor hiccups/stuttering i've been noticing in the delta motion?
The correlation to the network loop time is too close to be coincidental...
Is this abnormal?
I added the Signal strength since atheory has already been brewing in my head that:
Weak signal -> Network Lags -> CPU lags -> heater control loop losing control -> temperature excursion...

What do you think?

Thanks for your time and ideas!

Full M122:

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
Board ID: 0JD0M-9K662-MGPSS-6J1FD-3S46N-TVU6Y
Used output buffers: 3 of 24 (18 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93124 of which 48 recycled
Exception stack ram used: 592
Never used ram: 9328
Tasks: NETWORK(ready,308) HEAT(blocked,768) MAIN(running,1732) IDLE(ready,80)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 06:06:23 ago, cause: software
Last software reset at 2020-10-10 13:09, reason: User, spinning module GCodes, available RAM 9424 bytes (slot 2)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 0
MCU temperature: min 29.3, current 38.1, max 43.7
Supply voltage: min 23.7, current 24.1, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 2/423
Driver 1: standstill, SG min/max 13/423
Driver 2: standstill, SG min/max 41/408
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2020-10-10 19:16:18
Cache data hit count 4294967295
Slowest loop: 581.48ms; fastest: 0.12ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 2.9ms, write time 62.5ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 137, MaxWait: 6600918ms
Bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 2, completed moves: 2, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 581.68ms; fastest: 0.00ms
Responder states: HTTP(2) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address 24:62:ab:15:71:64
WiFi Vcc 3.39, reset reason Unknown
WiFi flash size 4194304, free heap 22440
WiFi IP address 192.168.1.6
WiFi signal strength -72dBm, reconnections 0, sleep mode modem
Socket states: 0 4 0 0 0 0 0 0

Phaedrux · 10 Oct 2020, 17:45

When do you see stutters in motion? Can you post your full config.g? Your M122 doesn't indicate any actual hiccup interruptions, but maybe it was gathered after a reboot?

I think this is still an issue of thermistor interference/noise.

Leav · 10 Oct 2020, 17:56

@Phaedrux Config files below.

Regarding wiring issue... it could be, I do understand that wiring issues can be tricky to replicate, but for the record I did try to heat up the extruder and jiggle the wires to see if I can create any noise, and no dice.

Config.g

; Configuration file for Duet WiFi (firmware version 3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.1.4 on Sun Jul 19 2020 22:30:32 GMT+0300 (Israel Daylight Time)

; General preferences
G90                                            ; send absolute coordinates...
M83                                            ; ...but relative extruder moves
M550 P"Goliath"                                ; set printer name
M665 R172.66 L511.07 B150 H329.8                 ; Set delta radius, diagonal rod length, printable radius and homed height
M666 X0 Y0 Z0                                  ; put your endstop adjustments here, or let auto calibration find them

; Network
M552 S1                                        ; enable network
M586 P0 S1                                     ; enable HTTP
M586 P1 S0                                     ; disable FTP
M586 P2 S0                                     ; disable Telnet

; Drives
M569 P0 S0                                     ; physical drive 0 goes backwards
M569 P1 S1                                     ; physical drive 1 goes forwards
M569 P2 S0                                     ; physical drive 2 goes backwards
M569 P3 S1                                     ; physical drive 3 goes forwards
M584 X0 Y1 Z2 E3                               ; set drive mapping
M350 X16 Y16 Z16 E16 I1                        ; configure microstepping with interpolation
M92 X100.00 Y100.00 Z100.00 E137.82            ; set steps per mm (E is 430 for bondtech, 137.82 for creality)
M566 X600.00 Y600.00 Z600.00 E1200.00          ; set maximum instantaneous speed changes (mm/min) (was [x,y,z]=1200)
M203 X9000.00 Y9000.00 Z9000.00 E1200.00       ; set maximum speeds (mm/min) (was [x,y,z]=18000)
M201 X500.00 Y500.00 Z500.00 E1000.00       ; set accelerations (mm/s^2) (was [x,y,z]=1000)
M906 X1000 Y1000 Z1000 E1500 I30                ; set motor currents (mA) and motor idle factor in per cent
M84 S30                                        ; Set idle timeout

; Axis Limits
M208 Z0 S1                                     ; set minimum Z

; Endstops
M574 X2 S1 P"!xstop"                           ; configure active-high endstop for high end on X via pin !xstop
M574 Y2 S1 P"!ystop"                           ; configure active-high endstop for high end on Y via pin !ystop
M574 Z2 S1 P"!zstop"                           ; configure active-high endstop for high end on Z via pin !zstop

; Z-Probe
M558 P5 C"!zprobe.in" H5 F250 T6000 ;A3 S0.01   ; set Z probe type to switch and the dive height + speeds (was 120mm/sec)
G31 P500 X30.31 Y17.5 Z2.44                    ; set Z probe trigger value, offset and trigger height (z probe is at a circle with radius 35mm from extruder)
M557 R160 S20									;Set probing radius to 160mm with a probe spacing of 20mm

; Heaters
M308 S0 P"bedtemp" Y"thermistor" T100000 B4725 C7.060000e-8 ; configure sensor 0 as thermistor on pin bedtemp
M950 H0 C"bedheat" T0                          ; create bed heater output on bedheat and map it to sensor 0
M307 H0 B1 S1.00                               ; enable bang-bang mode for the bed heater and set PWM limit
M140 H0                                        ; map heated bed to heater 0
M143 H0 S135                                   ; set temperature limit for heater 0 to 135C

M308 S1 P"e0temp" Y"thermistor" T100000 B4725 C7.060000e-8  ; configure sensor 1 as thermistor on pin e0temp
M950 H1 C"e0heat" T1                           ; create nozzle heater output on e0heat and map it to sensor 1
M307 H1 B0 S1.00                               ; disable bang-bang mode for heater  and set PWM limit

M308 S2 P"e1temp" Y"thermistor" T100000 B4725 C7.060000e-8  ; configure sensor 2 as thermistor on pin e1temp (chamber thermistor)


; Fans
M950 F0 C"fan0" Q100                           ; create fan 0 on pin fan0 and set its frequency
M106 P0 X1 H1 T45                            ; set fan H1 (was 0.25) value. Thermostatic control is turned on

; Tools
M563 P0 S"Extruder0" D0 H1 F0                  ; define tool 0
G10 P0 X0 Y0 Z0                                ; set tool 0 axis offsets
G10 P0 R0 S0                                   ; set initial tool 0 active and standby temperatures to 0C

; Lighting
M950 P0 C"e1heat"							   ; create output port 0 attached to heater 1\
M42 P0 S0.1									; Turn on Case light 10%


; Load Custom settings
M501

Config-override.g

; config-override.g file generated in response to M500 at 2020-10-08 22:54
; This is a system-generated file - do not edit
; Delta parameters
M665 L511.070:511.070:511.070 R173.652 H337.845 B160.0 X0.728 Y1.338 Z0.000
M666 X-1.091 Y-0.667 Z1.759 A0.00 B0.00
; Heater model parameters
M307 H0 A340.0 C200.0 D5.5 S1.00 V0.0 B1
M307 H1 A561.3 C290.3 D4.1 S1.00 V24.0 B0
; Workplace coordinates
G10 L2 P1 X0.00 Y0.00 Z0.00
G10 L2 P2 X0.00 Y0.00 Z0.00
G10 L2 P3 X0.00 Y0.00 Z0.00
G10 L2 P4 X0.00 Y0.00 Z0.00
G10 L2 P5 X0.00 Y0.00 Z0.00
G10 L2 P6 X0.00 Y0.00 Z0.00
G10 L2 P7 X0.00 Y0.00 Z0.00
G10 L2 P8 X0.00 Y0.00 Z0.00
G10 L2 P9 X0.00 Y0.00 Z0.00

Phaedrux · 10 Oct 2020, 17:59

@Leav said in Heater Temperature Excursion:

I did try to heat up the extruder and jiggle the wires to see if I can create any noise, and no dice.

Is this a direct drive extruder setup? As in, do the motor wires run along with the heater and thermistor and fan wiring?

Veti · 10 Oct 2020, 18:00

@Phaedrux
it looks like a delta.

please post a picture of the printer

Leav · 10 Oct 2020, 18:33

@Phaedrux @Veti

Here's the beast I call Goliath, a labor of blood, sweat and love

I've added some annotations to help understand how the wiring is handled. generally I tried to avoid putting sensor wires next to potentially noisy power wires, where possible.

Leav · 10 Oct 2020, 19:00

@Phaedrux said in Heater Temperature Excursion:

When do you see stutters in motion?

Forgot to answer this. Usually during long sweeping curves I will see the printer pause for a fraction of a second and continue.

I thought perhaps the CPU is getting overloaded...

Phaedrux · 10 Oct 2020, 19:37

When you notice it doing that check the m122 for hiccups. Though that's usually only when using microsteps higher than 16 which you are not. Curves with small segments could do it. What slicer do you use? s3d has been known to do that.

Low e jerk and pressure advance could be another possibility.

Do you use mesh compensation? Low z jerk can sometimes cause stutters in xy but you've got a delta so all jerk is the same on xyz so I don't think that's what's going on here.

Leav · 10 Oct 2020, 19:43

@Phaedrux said in Heater Temperature Excursion:

When you notice it doing that check the m122 for hiccups. Though that's usually only when using microsteps higher than 16 which you are not. Curves with small segments could do it. What slicer do you use? s3d has been known to do that.

I use PrusaSlicer, will do the M122 when I notice the hiccups.

Low e jerk and pressure advance could be another possibility.

Interesting, I don't use pressure advance but I can see why the low E jerk could cause that.

Do you use mesh compensation? Low z jerk can sometimes cause stutters in xy but you've got a delta so all jerk is the same on xyz so I don't think that's what's going on here.

Yes I'm definitely a mesh user. I see how that could cause issues for cartesian users, but I agree probably unrelated for me on a delta.

If I may re-ask the question from a few posts ago: what do you make of the "slowest loop" statistic? Is that not concerning?

Phaedrux · 10 Oct 2020, 19:50

Unsure of the slowest loop stat. @dc42 would have to comment.

Leav · 13 Oct 2020, 06:03

I hope this isn't bad form, but it seems this question slipped by so I'm tagging @dc42 again...

Phaedrux · 13 Oct 2020, 06:29

@Leav said in Heater Temperature Excursion:

I hope this isn't bad form, but it seems this question slipped by so I'm tagging @dc42 again...

No that's fine. Keeps us from forgetting. Thanks for giving it a couple days.

Leav · 17 Oct 2020, 10:04

Still chasing this, and I've worked on a few things since I last posted:

New Thermistor (cartridge style, to rule out issues with wires shorting in the heater block)
New heater cartridge
Successfully replaced wifi module with esp-07s - Signal went from ~-70dbm to ~-55dbm

Started a five hour print, temperature graph was smooth, minus a few blips which seemed to indicate that the heating process gets "stuck" (??) either in "on" or "off" mode for a second, and the then recovers:

I let it be, thinking this is a minor issue, but after a few hours the excursion hit again:

I was able to resume the print (thanks for such a robust recovery mechanism!), and there was still a bit of noise:

But now it is very smooth (minor blip still shows up):

Does anyone have any clue what could cause this? I feel like this is some sort of random behavior that gets introduced and suddenly vanishes, making debugging very frustrating...

Here is a copy of the M122 response after the recovery:

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
Board ID: 0JD0M-9K662-MGPSS-6J1FD-3S46N-TVU6Y
Used output buffers: 3 of 24 (11 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93124 of which 48 recycled
Exception stack ram used: 576
Never used ram: 9344
Tasks: NETWORK(ready,268) HEAT(blocked,792) MAIN(running,1732) IDLE(ready,80)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 04:52:37 ago, cause: power up
Last software reset at 2020-10-16 16:27, reason: User, spinning module GCodes, available RAM 9376 bytes (slot 0)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 0
MCU temperature: min 22.3, current 36.8, max 43.8
Supply voltage: min 23.7, current 24.0, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: ok, SG min/max 42/430
Driver 1: ok, SG min/max 78/423
Driver 2: ok, SG min/max 0/416
Driver 3: ok, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2020-10-17 12:48:30
Cache data hit count 4294967295
Slowest loop: 275.25ms; fastest: 0.12ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 2.9ms, write time 63.4ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 161, MinFreeDm: 137, MaxWait: 455270ms
Bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 215624, completed moves: 215603, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: 3
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.4
=== GCodes ===
Segments left: 1
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X-5.074 Y-103.468 E29.61163" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 275.44ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address 48:3f:da:45:74:de
WiFi Vcc 3.38, reset reason Unknown
WiFi flash size 4194304, free heap 24504
WiFi IP address 192.168.1.6
WiFi signal strength -55dBm, reconnections 0, sleep mode modem
Socket states: 4 0 0 0 0 0 0 0