Results of Testing Heater Fault Interlocks / PS_ON

DocTrucker

Evening all,

Preliminary results from using variable resistors to fake thermistor input to test faults.

Heater fault detection configured in config.g as follows:

M143 H0 S120 A0 C0    ; Raise a heater fault if heatbed exceeds 120C.
M143 H1 S255  A0 C0   ; Raise a heater fault if hotend exceeds 255C.
M143 H0 S10 A0 C1     ; Raise a heater fault if heatbed falls below 10C.
M143 H1 S10  A0 C1    ; Raise a heater fault if hotend falls below 10C.
M570 H0 P10 T10 S1    ; Allow a heat bed anomaly to persist for 10 seconds (P10) 
                      ; on the before raising a heater fault. Allow a 10C 
                      ; deviation from set point (T10) After 1 minutes of heater 
                      ; fault cancel the build (S1).
M570 H1 P10 T10 S1    ; Allow a hot end anomaly to persist for 10 seconds (P10)
                      ; on the before raising a heater fault. Allow a 
                      ; 10C deviation from set point (T10) After 1  
                      ; minute of heater fault cancel the build (S1).

The test board was set to fake a 50C set point for the bed, and 5 selectable values for hotend temp of 5, 195, 210 (process set point), 225, and 260C.

Notes:

The ten second delay appears to only apply to the temp deviation, not the minimum and maximum temperature limits.
Exceeding upper temp limit has a different delay before action than lower.
Builds can be started when the reported temperature exceeds the allowable deviation and the machine will not fault.
Builds can be started when reported temp is above maximum and the system will never fault.
In last two cases the system will fault if the temp sensor goes open circuit.
The configured delay before cancelling a build is applied to exceeding allowable deviation from set point, or exceeding upper and lower limits.
A noisy temp signal will instantly cause issues with extrusion. The delay before fault is not considered.

Advised actions:

Ensure set delay before faults is respected by upper and lower limit trips.
Allow use of decimals on minute delay before cancel builds.
If temperature is above set point more than the permissible deviation ensure the timer starts to trip fault.
Prevent build from starting if reported temp is above maximum, or fault out after timers.

Testing on generation 1 duet v0.6 with 1.26.1 firmware. Relays controlled by PS_ON.

Related threads:
Configuring heater fault detection PS_ON trip
PS_ON Safety question
PS_ON Pin what can control it
Adding a 24v safety relay for the heaters any thoughts

More detail to follow - probably tomorrow

DocTrucker

Here's the report video. Bit of a monster and only a draft - so full of typos! I'll add some time stamps for the main bits but essentially there is:

Experimental setup.
Tests with no delay between heater fault and build cancel.
Tests with a minute delay.
Tests starting the process too hot.

https://youtu.be/c80-CREaZLc

DaBit

I will probably have to watch the video which i didn't yet, but why such an old firmware?

Phaedrux

@DaBit said in Results of Testing Heater Fault Interlocks / PS_ON:

but why such an old firmware?

It's actually quite recent despite the version number. It's a back ported version for the older Duet boards.

DocTrucker

@DaBit Already stated by @Phaedrux but the board is Duet v0.6. I've a Duet3 machine in build and will test that in the same way too. Either way Duet v0.6 was still being sold by RepRapLtd up until last year, so there are many about and it knocks spots of the RAMPS setups for usability and so a second hand v0.6 is still an attractive proposition for cost limited opperations like mine.

Main issues are the builds can be started over the maximum temperature (and no fault is raised) and the cold extrusion lock comes in the instant of a low temp reading and so sensor noise is likely to create marks on the part.

DocTrucker

@T3P3Tony @dc42 @droftarts Can you confirm if the three main bugs identified here have been resolved in RRF3?

Starting and running a build process when temp is reading above maximum with no fault being raised.
Starting and running a build process when temp is reading above the allowable deviation with no fault being raised.
Noisy temp signals causing instant cold extrusion locks rather than needing to persist for a configurable time first.

...if not please can they be added to the work/test list.

droftarts

@DocTrucker the first two relate to M143 and were fixed in 3.0.1 RC2, see https://github.com/dc42/RepRapFirmware/blob/v3-dev/WHATS_NEW_RRF3.md#reprapfirmware-301-rc2

Don’t know about the last one. Probably better to fix the noisy reading rather than hide a potential failure with firmware hack!

Ian

DocTrucker

@droftarts well that potential failure is already hiden with the time parameter of M570 and so not consistant? Regardless the PT100 sensors (yes I realise I'm not testing PT100) are known to be noisy and a system can only get dangerous in a under a handful of seconds if you are running a massively over-powered heater.

DocTrucker

@droftarts Thanks (being genuine here, not sarcastic just to be clear ) for the reply anyway!

The bug fix you linked to appears to be about setting an extruder temperature - M109 - rather than starting a process when the system reports high temps with the set temp lower. I was anle to start processes with temp set to 210C and the system reporting 225C and 260C above both the deviation and upper limit faults. Machine cartied on until I stopped it.

droftarts

@DocTrucker firmware version you’re running? M143 setting for the heater you set? Steps in the ‘process’ to create the fault? If you want us to be able to replicate and troubleshoot this, you need to be specific.

Ian

DocTrucker

@droftarts have you looked at my first post and the 17 minute video showing full setup, tests and results? $:-\$

DocTrucker

@droftarts how would you like me to be more specific and thorough?

droftarts

@DocTrucker sorry, phone screen wasn’t showing early posts. You’ve been more than thorough!

Ian

droftarts

@DocTrucker as you’re using Duet 0.6 and RRF 1.26.1, we’ll have to wait for @dc42 to say how further support/firmware changes for older boards and RRF 1/2 are going to be implemented.

Ian

dc42

I announced back in mid 2017 that I was stopping support for Duet 06 and 085 (see for example https://forum.duet3d.com/post/21969). I have never received any financial benefit from the sale of those older Duets. In fact I continued to do builds for Duet 06 and 085 for another two and a half years, although I relied on others to test them. But I have to draw the line somewhere.

Of course it is open to anyone to back-port changes in RRF3 to RRF2 where they are relevant.

DocTrucker

@droftarts no worries.

DocTrucker

@dc42 I understand you rely on others to test them. That is what I have done, and shared the results with you. However, my specific question to the Duet staff was; "Can you confirm if the three main bugs identified here have been resolved in RRF3?" ...as I suspected these bugs may have persisted through the newer versions of RRF. Basically a heads up to check for some odd behaviours.

zapta

@DocTrucker, can you elaborate on the PS_ON aspects of your experiments? For example, did the PS_ON output changed state on heater fault? Did you have any PS_ON specific configuration? (the documentation of the M570 command doesn't mention any PS_ON configuration). The video doesn't seem to have a visual indication of the state of PS_ON.

[if PS_ON reacts automatically to heater fault, I would like to reproduce it on my machine and then use it to cut heaters power using a mechanical relay]

DocTrucker

@zapta Sure. I was running a four channel Arduino mechanical relay board for these tests, which was driven by PS_ON. I didn't need to configure this in an special way other than putting M80 as the last line of the config.

M570 H0 P10 T10 S0    ; Allow a heat bed anomaly to persist for 10 seconds (P10) 
                      ; on the before raising a heater fault. Allow a 10C 
                      ; deviation from set point (T10) After 0 minutes of heater 
                      ; fault cancel the build (S0).
M570 H1 P10 T10 S0    ; Allow a hot end anomaly to persist for 10 seconds (P10)
                      ; on the before raising a heater fault. Allow a 
                      ; 10C deviation from set point (T10) After 0  
                      ; minute of heater fault cancel the build (S0).

With the heater fault detection setup as above the PS_ON was triggered as the build process was cancelled. The S parameter controlled the time between the heater fault bring raised and the build being cancelled. I tried setting it to 0.17 (10 seconds) but that was just treated as 0. I assume this input is integer, but to be fair I didn't try decimals above 1. With 1 as expected the delay between fault raised and PS_ON interlock was 1 minute.

The upper and lower deviations in the above example allow the temerature to deviate by 10 degrees from set point without error. It also states that the temp can exceed this range for 10 seconds before tripping out. I assumed this is to hrlp in situations like the part cooling fan reflecting of a part and crash cooling the head, or electrical noise. The 10 second parameter behaved as expected for the deviation faults, but not for the upper/lower limit alarms.

The responce time for the exceefing maximum/minimum limits appeared a little inconsistent, but was generally close to instantly raising the fault, before waiting with the fault for the time defined in the S parameter of M570 before PS_ON interlock.

The heater not warming as quickly as expected alarm behaved similarly to the exceeding upper or lower limits alarm.

Essentially all behaved broadly as expected with regards the PS_ON control of a mechanical relay. The only surprises were you could cause the upper deviation and upper limit alarms to fail to trigger by starting a build process above these points. There was also indication that semseor noise is likely to cause artefacts on printed parts as dropping low causes the extruder to lock.

With my relays they were rated high enough to interlock the posative feed directly to the heaters. This is the least fuss way of interlocking as you need not worry about fans. If you interlock the mains side of the VIN supply then you need an additional 5V supply to the board, and should consider a seperate supply for the fans too or a heater interlock could wuickly result in a jammed hotend.

Post written from phone, so many typos expected! I'll correct thrm as I spot them!

DocTrucker

I've got a Duet 3 and will run these tests as soon as practical (machine not ready yet) as it appears the exact behaviour of RRF3 is not clear. I would assume it is no worse than RRF1, but tests will verify.