Heater fault checking routine to be run in daemon.g

OwenD · 5 Jan 2021, 12:48

This is a routine I run in daemon.g to check for heater faults.
It is mainly aimed at covering the time when the printer is idle as RRF already checks when a print is running.
I have excluded heaters that are currently being PID auto tuned
It does not take action if the heater temp is < 45 degrees but will shut down if temp is < zero as that would indicate a thermistor fault.
If the temperature is over 45 degrees it will only fault if the current temp is > 15 degrees from the set active temp AND the temperature is rising by > 0.5 degrees over 3 seconds.
This is to avoid activating when the temp is deliberately lowered.
NOTE: I have used a dummy tool to store variables which allow me to have this routine run only at intervals greater than 10 seconds whilst allowing the rest of daemon.g to run at normal intervals. Without this (and until we get variables) you will need to modify the code and use G4 to set the timing
For the timing I have used state.upTime. which will probably roll over at some point if the printer is left on long enough (I'm guessing it's a 16 bit signed integer?), so I have a check for that.
I have tested it in both idle and printing states and it seems to work as expected but have not confirmed the upTime rollover works as expected.

; 0:/sys/daemon.g
; runs continually in background at approximately 1Hz if not delayed internally


;HEATER CHECKS

; this section of daemon.g checks for heater faults
; RRF doesn't currently check for faults when idle but by default will shutdown during printing if temperature excursion is > 15 degrees.
; Note: temp excursion value and time  may be manually set using M570
; checking if temp is rising requires a variable.  I have used a dummy tool offset (Tool2)  for this and to do elapsed time since last check
; G4 could be used but would also delay anything else in daemon.g
; this way allows other checks to run more frequently if needed however the G4 delays inside the loop will affect the frequency of daemon.g
; will be updated when variables are available in RRF.
while iterations < #heat.heaters ; loop through all configured heaters
	if ((tools[2].offsets[1]+10) > state.upTime) ; if checked in last 10 seconds escape loop and go to rest of daemon.g if present.  offset will be zero at startup via config.g
		;echo "skipping loop " ^ " " ^ state.upTime  ^ " " ^ tools[2].offsets[1]+10
		if tools[2].offsets[1] - state.upTime > 60 ; uptime must have rolled over so reset off set to zero
			G10 P2 Y0
			echo "upTime has rolled over.  Heater checking reset"
		break
	;echo "checking heater " ^ iterations ^ " " ^ state.upTime  ^ " " ^ tools[2].offsets[1]+10
	if heat.heaters[iterations].state="tuning"
		;echo "heater " ^ iterations ^ " is tuning - no check carried out"
		continue ; don't check this heater as it is PID auto tuning
	if (heat.heaters[iterations].current) > (heat.heaters[iterations].max) ; temp is over max so emergency shutdown required
		;M41 P5 S1  ; activate output connected to externally powered latching relay here to sound alarm
		echo "heater over max temp fault detected in daemon.g.  - shutting down"
		M112; emergency shutdown
		M81 S1 ; turn off power when fans have turned off
	if (heat.heaters[iterations].current > 45) ; no real danger at below this temp as ambient may be close to this
		;echo "heater " ^ iterations ^ " is above 45 degrees"
		if (heat.heaters[iterations].state!="off") && (heat.heaters[iterations].current > heat.heaters[iterations].active + 15) ; temp is > 15 degrees above target.
		;echo "heater " ^ iterations ^ " is on or in standby - checking if temp is rising"
			G10 P2 X{heat.heaters[iterations].current} ; set dummy tool X offset to current temp of heater
			G4 S3 ; wait 3 seconds
			if (heat.heaters[iterations].current > tools[2].offsets[0] + 0.5) ; heat is rising by more than 0.5 degrees in 3 seconds
				echo "heater runaway fault detected in daemon.g.  - shutting down"
				if (state.status=="processing")
					M25 ; pause print so you might be able to save it using M119
				;M41 P5 S1  ; activate output connected to externally powered latching relay here to sound alarm
				M0 ; unconditional stop.  If axes are homed and a print is being canceled will run cancel.g  otherwise will run stop.g
				M81 S1 ; turn off power when fans have turned off
			else
				;echo "heater is on or standby but temp is falling on heater " ^ iterations ^ " - no action needed"
		elif (heat.heaters[iterations].state="off") && (heat.heaters[iterations].current >= 45) ; if heater is off and temp is greater than 45 there could be an issue
			;echo "heater " ^ iterations ^ " is off but checking if temp is rising"
			G10 P2 X{heat.heaters[iterations].current} ; set dummy tool X offset to current temp of heater
			G4 S3 ; wait 3 seconds
			if (heat.heaters[iterations].current > tools[2].offsets[0] + 0.5) ; heat is rising by more than 0.5 degrees in 3 seconds
				;echo "heater is off but temp is rising on heater " ^ iterations ^ "emergency shutdown"
				;M41 P5 S1  ; activate output connected to externally powered latching relay here to sound alarm
				echo "heater runaway fault detected in daemon.g.  - shutting down"
				M112; emergency shutdown
				M81 S1 ; turn off power when fans have turned off
			else
				;echo "heater is off & temp is falling or stable on heater " ^ iterations ^ " - no action needed"
	else
		;heater is below 45 degrees so only other fault may be an open circuit thermistor which should show -275 degrees
		if heat.heaters[iterations].current < -0 ; we probably have a thermistor fault if heater is less than 0 degrees
			M112 ; emergency shutdown
			M81 S1 ; turn off power when fans have turned off

	if iterations == #heat.heaters-1 ; all heaters have been checked
		G10 P2 Y{state.upTime} ; set the new time to check again

; END HEATER CHECKS


; run other checks

;G4 S10 ; pause daemon.g for 10 seconds before next run - required if not using variables

garethky · 8 Jan 2021, 05:55

Nice work!

I ran into this mid print yesterday. I got about 1 hour and 30 minutes into the print. Herd a loud clicking coming from the extruder. Took a few minutes to figure out why! The heater had faulted and shut off, but the print hadn't stopped!

I really wish we didn't have to write a routine for this. A fault mid job should stop or pause the job and log an error to the console. Maybe there are some janky printers out there that have intermittent faults that this would break, but come on!

What part of the model does the GUI read to get the "fault" that shows up on the heater?

OwenD · 16 Oct 2021, 05:48

Someone asked about this macro, so I thought I should post the latest version of it that I am running as we now have variables.
Note that as I'm using a while true loop, daemon.g will only load one and keep looping.
Hence I have some variables to shut off sections.

So in config.g I have

; Global variable to run/not run daemon.g - checked in daemon.g and abort if false
if !exists(global.RunDaemon)
	global RunDaemon = true  
else
	set global.RunDaemon = true 

; Global variable to run/not run heater checks daemon.g - checked in daemon.g and abort if false
if !exists(global.RunHeaterChecks)
	global RunHeaterChecks = true  
else
	set global.RunHeaterChecks = true
while heat.heaters[1].current=2000 ; loop until thermistor values are stable
	G4 P1
	if iterations > 10000 ; if it takes more than 10 seconds we have a problem with the thermistor
		M118 P0 L1 S"Thermistor failed to stabilize in less than 10 seconds"
		break
;echo "sensor stable time: " ^ state.upTime ^ "." ^ state.msUpTime

if !exists(global.LastTemp) || global.LastTemp=null
	global LastTemp=heat.heaters[1].current ; Set variable to current extruder temp.
else
	set global.LastTemp=heat.heaters[1].current ; Set variable to current extruder temp.
G4 P10
if !exists(global.LastCheckTime)
	global LastCheckTime=0 ; variable for use in daemon.g 
else
	set global.LastCheckTime=0 ; variable for use in daemon.g

if !exists(global.filamentDistance)
	global filamentDistance = 0 ; global for use to allow filament to feed for set distance after sensor trips 
else
	set global.filamentDistance = 0

In Daemon.g I have

; 0:/sys/daemon.g
; runs continually in background at approximately 10 second intervals
; We have initiated an infinite loop at the start so that we can do things at intervals less than 10 seconds
; daemon.g won't be opened again if it it still running
; everything must be indented by 2 x tabs becasue of the infinite loop at the start to allow checks at intervals less than 10 seconds
; We have created a global variable in config.g called RunDaemon
; If RunDaemon is set to "false" we exit the daemon without running anything.  However daemon.g will still try to run every 10 seconds if it exists.


if global.RunDaemon == false
	;echo "exiting daemon"
	M99 ; don't run the daemon

;HEATER CHECKS

; this section of daemon.g checks for heater faults
; RRF doesn't currently check for faults when idle but by default will shutdown during printing if temperature excursion is > 15 degrees.
; Note: temp excursion value and time  may be manually set using M570
; checking if temp is rising requires a variable.
; G4 could be used but would also delay anything else in daemon.g
; this way allows other checks to run more frequently if needed however the G4 delays inside the loop will affect the frequency of daemon.g
; We are only checking the heaters every 10 seconds.  This can be varied by changiing the global variable global.HeaterCheckInterval
; if the global variable RunHeaterChecks is false, then we skip this section but still run the remainder of the daemon

while true
	if (global.RunHeaterChecks == true) && (global.RunDaemon==true)
		if state.upTime < 60
			break; If uptime is < 60 seconds, break out so all fans etc have time to stabilise.

		if ((global.LastCheckTime + global.HeaterCheckInterval) > state.upTime) ; if checked in last 10 seconds escape loop and go to rest of daemon.g if present.  offset will be zero at startup via config.g
			;echo "skipping loop " ^ " " ^ state.upTime  ^ " " ^ global.LastCheckTime+global.HeaterCheckInterval
			if global.LastCheckTime-state.upTime > 60 ; uptime must have rolled over so reset off set to zero
				G10 P2 Y0
				echo "upTime has rolled over.  Heater checking reset"
				break
		else
			;add code that must be run at less than 10 second intervals here
			while iterations < #heat.heaters ; loop through all xconfigured heaters
				set global.LastTemp=heat.heaters[1].current ; Set variable to current extruder temp.
				if heat.heaters[iterations].state="tuning"
					;echo "heater " ^ iterations ^ " is tuning - no check carried out"
					continue ; don't check this heater as it is PID auto tuning
				if (heat.heaters[iterations].current) > (heat.heaters[iterations].max) ; temp is over max so emergency shutdown required
					;M41 P5 S1  ; activate output connected to externally powered latching relay here to sound alarm
					M118 P0 S"heater over max temp fault detected in daemon.g.  - shutting down" L1
					M112; emergency shutdown
					M81 S1 ; turn off power when fans have turned off
				if (heat.heaters[iterations].current > 45)  &&  (heat.heaters[iterations].active > 45); no real danger at below this temp as ambient may be close to this
					;echo "heater " ^ iterations ^ " is above 45 degrees"
					if (heat.heaters[iterations].state!="off") && (heat.heaters[iterations].current > heat.heaters[iterations].active + 15) ; temp is > 15 degrees above target.
						echo "heater " ^ iterations ^ " is on or in standby - checking if temp is rising"
						set global.LastTemp=heat.heaters[iterations].current ; set the last check temp
						echo "heater " ^ iterations ^ " temp: " ^ heat.heaters[iterations].current
						G4 S3 ; wait 3 seconds
						if (heat.heaters[iterations].current > global.LastTemp + 0.5) ; heat is rising by more than 0.5 degrees in 3 seconds
							echo "heater runaway fault detected in daemon.g.  - shutting down"
							if (state.status=="processing")
								M25 ; pause print so you might be able to save it using M119
							;M41 P5 S1  ; activate output connected to externally powered latching relay here to sound alarm
							M0 ; unconditional stop.  If axes are homed and a print is being canceled will run cancel.g  otherwise will run stop.g
							M81 S1 ; turn off power when fans have turned off
						else
							;echo "heater is on or standby but temp is falling on heater " ^ iterations ^ " - no action needed"
					elif (heat.heaters[iterations].state="off") && ((heat.heaters[iterations].current) >= (fans[1].thermostatic.lowTemperature+0)) ; if heater is off and temp is greater than 50 there could be an issue
						set global.LastTemp=heat.heaters[iterations].current;
						;echo "heater " ^ iterations ^ " is off but checking if temp is rising"
						G4 S3 ; wait 3 seconds
						if (heat.heaters[iterations].current > global.LastTemp + 0.5) ; heat is rising by more than 0.5 degrees in 3 seconds
							echo "heater is off but temp is rising on heater " ^ iterations ^ "emergency shutdown"
							;M41 P5 S1  ; activate output connected to externally powered latching relay here to sound alarm
							echo "heater runaway fault detected in daemon.g.  - shutting down"
							M112; emergency shutdown
							M81 S1 ; turn off power when fans have turned off
						else
							;echo "heater " ^ iterations ^ " is off & temp is falling or stable on heater " ^ iterations ^ " - no action needed"
				else
					;echo "heater " ^ iterations ^ " is below 45 degrees so check thermistor" ;"heater is below 45 degrees so only other fault may be an open circuit thermistor which should show -275 degrees"
					if heat.heaters[iterations].current < 0 ; we probably have a thermistor fault if heater is less than 0 degrees
						M112 ; emergency shutdown
						M81 S1 ; turn off power when fans have turned off

				;Check if water pump is running correctly
				if (iterations=1) && ((heat.heaters[1].current) > (fans[1].thermostatic.lowTemperature+0))
					if fans[1].rpm <= 500 ; Coolant pump RPM off or low
						G4 S3 ; check again in 3 seconds in case it's just spinning up
						if fans[1].rpm <= 500
							echo "Water pump fault - shutting down heaters - RPM : " ^ fans[1].rpm
							M25 ; pause print so you might be able to save it using M119
							M0 ; unconditional stop.  If axes are homed and a print is being canceled will run cancel.g  otherwise will run stop.g
							M81 S1 ; turn off power when fans have turned off
					elif (fans[1].rpm > 500) && (fans[1].rpm < 1400)
						G4 S3 ; check again in 3 seconds in case it's just spinning up
							if (fans[1].rpm > 500) && (fans[1].rpm < 1400)
							echo "WARNING: Water pump RPM low - RPM : " ^ fans[1].rpm
							if (state.status=="processing")
								M25 ; pause print so you might be able to save it using M119
								;M41 P5 S1  ; activate output connected to externally powered latching relay here to sound alarm
								M0 ; unconditional stop.  If axes are homed and a print is being canceled will run cancel.g  otherwise will run stop.g
								M81 S1 ; turn off power when fans have turned off
					else
						;echo "Coolant OK - RPM : "  ^ fans[1].rpm

					if fans[2].rpm <=1000
						echo "WARNING: Water pump FAN RPM low - RPM : " ^ fans[2].rpm
						if (state.status=="processing")
							M25 ; pause print so you might be able to save it using M119
							;M41 P5 S1  ; activate output connected to externally powered latching relay here to sound alarm
							M0 ; unconditional stop.  If axes are homed and a print is being canceled will run cancel.g  otherwise will run stop.g
							M81 S1 ; turn off power when fans have turned off
					else
						;echo "water pump fan OK"

				if iterations == #heat.heaters-1 ; all heaters have been checked
					set global.LastCheckTime=state.upTime ; set the new time to check
					;echo "new check time = " ^ global.LastCheckTime
	; END HEATER CHECKS	


			
	; BEGIN OTHER CHECKS AT INTERVALS LESS THAN 10 SECONDS.  

	; This should only trigger if a print is in progress and the runout value has been set by a runout.  NOTE: Does not work s at RRF3.4b5 as a filament error causes immediate pause


	if (move.extruders[tools[state.currentTool].filamentExtruder].position > global.filamentDistance)  &&  (global.filamentDistance !=0) && (state.status = "processing")
		echo "paused called from daemon"	
		M25 ; pause print if filament has run out

        ; DO SOME OTHER CHECKS
	G4 S2 ; add a delay for these checks