Both hot end heaters go "offline" concurrently during a print

deckingman

@jay_s_uk said in Both hot end heaters go "offline" concurrently during a print:

@deckingman I think you're going to have to bite the bullet then and go for a 3.4 upgrade.

I think you are right. But before I do that, I've noticed something in my config.g which might have a bearing. I recently changed the way I set tool temperatures from using G10 S and R to the newer M568 method. Trawling through my config.g file, I noticed that for tools 0 and 1, there was no "A" parameter but for tools 2 to 5, there was an "A0". Which means the initial state was set to "off" for those tools. Now it shouldn't matter because the tools get set to active when selected.

But up until now, I've only been using tools 0 and 1 (PLA and ABS) which have no "A" parameter and so the initial state is "Active". These latest prints were the first with ASA which is tool 4 and which had an initial state (on boot) of "Off". So there is a chance that I've unearthed a bug which is related to using M568 and the A0 parameter.

On the basis that I only like to change one thing at a time, I've removed the "A0" parameter for all tools. I'll try a print (the same one) like that and see what happens. If the fault persists, I'll upgrade the firmware and try again.

jay_s_uk

@deckingman looking forward to seeing the results of that. I don't think many people use M568 yet

deckingman

@jay_s_uk So, this next print has gone beyond the point where the other two prints failed, and thus far is continuing without any issues. Which means there are two possibilities.

There is a bug in the firmware related to using the A0 parameter with M568 in config.g - possibly where there are multiple tools defined, some of which use the A0 parameter while other tools do not.
This particular gremlin has had a good nights sleep, is feeling better, and has decided not to annoy me just yet - but may do so later..........

I'll run with this for now rather than updating the firmware to see if I get any reoccurrence. Maybe @dc42 could take a gander at the code and see if he can spot anything related to using A0 in M568 on some tools in a multiple tool setup (and two heaters common to all tools). Or maybe he could spot something in the M122s I put in that google drive folder.

A Former User

This post is deleted!

deckingman

So it seems that I spoke too soon. Some time after my last post, the heaters went offline again. I can confirm that the Can Diag light on the offending board was flashing in sync with all the other boards. I grabbed another set of M122s while the fault was present - before I aborted the print. I've uploaded it to that folder on my google drive. https://drive.google.com/drive/folders/114bAgLBiS9yps7O8-5aHD6ute3v_tiz4?usp=sharing. The particular file is entitled "Diagnostics 040522 While fault present"

Comparing the M122 results for the 3 expansion boards, board 1 (the one that the heater are attached to) has different and strange comments regarding software reset. Board one shows the last software rest time as yesterday (which doesn't make much sense to me) but the reason is "WatchdogTimeout, available RAM 159104, slot 0" followed by a bunch of reset codes and stack values which the other two boards do not show.

I'll update the firmware to the latest now, but in the meantime, perhaps DC42 could take a look at those diagnostic reports?

deckingman

@arnold_r_clark said in Both hot end heaters go "offline" concurrently during a print:

@jay_s_uk said in Both hot end heaters go "offline" concurrently during a print:

@deckingman I think you're going to have to bite the bullet then and go for a 3.4 upgrade.

I think you might find 3.4 is not any better at handling spurious heater "faults" than 3.3 is, I have been battling multiple random types of heater faults for a while thinking it has been hardware/wiring etc which i have changed only for the same random heater "faults" to pop up across multiple different machines, faults which can also magically disappear with a reboot which were previously listed as being a physical failure of a temperature measurement component.. which has then gone on to carry out 24hr+ prints with no hardware change...

Yes I've seen all those posts - that's what worries me about "updating" the firmware. Although, in my case, I'm not getting any fault messages - just the heaters go "offline" and because of that, they turn off. But the print continues (without any bloody filament being extruded).

But I've learned that "upgrading" to the latest firmware seems to be a prerequisite of getting any "official support". I expect I'll end up with heater faults as well as the original problem of them going offline............

deckingman

Firmware duly updated to 3.4 "stable". Problem with hot end heaters going "offline" during a print persista. Again, I ran M122 on all boards. File upload to the same google drive folder. File name is "Diagnotics with fault present and 3.4 firmware"

This time, board 1 is showing a "reset" (as opposed to a "software reset" - although the previous software reset from yesterday is also still present ). The full message is

"Last reset 00:00:17 ago, cause: core brownout"

I'm leaning towards suspecting this expansion board is bricked but I'd like confirmation from one of the Duet guys.

jay_s_uk

@deckingman that appears to be the brown out detector for the internal core voltage on the SAME5 MCU.

1.2V Brown-Out Detector (BOD12)
The BOD12 is calibrated in production and its calibration configuration is stored in the NVM User Row.
This configuration must not be changed to assure the correct behavior of the BOD12. The BOD12
generates a reset when 1.2V crosses below the preset brown-out level. The BOD12 is always disabled in
Standby, Hibernate, and Backup Sleep modes.

heres a datasheet https://www.mouser.com/datasheet/2/268/60001507A-1130176.pdf
either way, it doesn't seem to be something set by the firmware (as the datasheet explicitly states its calibrated at the factory and shouldn't be messed with) which would lean more towards a hardware than software issue.
Hopefully the duet guys will chip in.

deckingman

@jay_s_uk Thanks. That's two of us leaning towards it being a hardware issue. I'll wait until one of the Duet guys chips in. On the basis that everything else worked when board 1 reset due to a brown out, to get myself out a hole, I've moved the heaters to board 2. I'll see if I can get a print out of it like that.

I hope I can get it repaired or some other cheap fix. If not, that'll be the end of my multi-material exploits because there is no way that I can afford the £114 for a new board with the current cost of living crisis, and Boris breaking his promises on the pension triple lock.

deckingman

@jay_s_uk By way of an update, I got a print done (about 10 hrs) by moving the heaters from board 1 to board 2. So I'd say that's conclusive proof that board 1 has a hardware fault as we both suspected. Thanks for your input.

Phaedrux

@deckingman said in Both hot end heaters go "offline" concurrently during a print:

I'll wait until one of the Duet guys chips in.

We had seen it and were waiting for DC42 to take a look, and I see you've now been in touch.