Duet 2.05 memory leak?

kazolar

@dc42 i think we're going around in circles -- 2.5.1 likely would fix the software issue, but there is clearly a hardware fault -- i HAVE to let it run for 30+ minutes and fail before reset and start a series of prints that will succeed. That 30 minutes time frame -- maybe the time that the cold solder joint -- somewhere heats up things start running fine. This board was purchased on Aug 3 2018 -- Filastruder order #36936
It's either covered by warranty or it's not. If not I'll order a new one from filastruder.

I said that I was ready to go ahead and do this -- I was asked to slice this in different slicers and try different SD cards -- I have. I am printing face shields for hospital workers. I have a donation of 200 being picked to go to a local hospital tomorrow. I'd rather lose more sleep and waste time troubleshooting what at this point feels like a bad solder joint -- is it worth my time to take the board out look at under a scope and look for it -- it could be on the sd card connector -- another trace. I resolves itself after 30 minutes of printing. The question is at this point how is the replacement being handled.

kazolar

@dc42 what makes you think it's an SD card error still? I had the same underrun error reproduced with Ocotprint. I am using a branded SD card which tests out at the top of speed spectrums. The predictable behavior now is that after the machine has been either idle or off for some time, it takes 1 failed print due to an underrun --which happens around 30-40min (not same spot each time anymore) and then after a reset the next set of prints if I don't let it cool down work fine --so as some suggested a cold solder joint which ends up warming up and working fine after some heat up time is the culprit here. I'm going to wait for a response as to how to proceed -- If I don't get one -- I'll order a new duet 2 from filastruder.
@droftarts Considering the importance of these prints, and my past experience here, I rather disappointed and surprised with the level of support. YouTubers who do questionable videos (I can name some -- who I used to watch) get free boards sent to them, I guess I don't have that kind of a public image -- I will be releasing the build vlog of this printer, which has been long overdue as I've filmed it starting 3 years ago. The last few episodes will feature my experience with using this machine to print these shields. I am donating countless hours of my time, our group is getting daily requests from local hospitals -- and even those further away for ear relievers. I am kinda getting burn out mostly due the issues with the duet board. I was dealing with these behaviors initially -- and was willing to keep troubleshooting this. Now we're at a crossroads. I have played the -- do this, do that steps, which would be fine if we weren't in time crunch in a pandemic and every shield I produce is another doctor/nurse, EMT worker who won't get sick. I don't quite understand the lack of urgency here, or maybe you're not taking this seriously enough -- I had first hand accounts of doctors dying in New York, and young nurses succumbing to the illness in the Bronx.

bot

As for your first question directed at dc42: octoprint sometimes waits for an OK response from the firmware, and maybe for other reasons, the stuttering might occur on any setup. It doesn't rule out the SD card being a problem.

kazolar

@bot
So yes, an SD card problem may be there, which is likely a hardware issue since after 30-40 minutes of a failed print, the next print is fine. In fact the next 3-4 prints are fine, provided I don't let the board/system cool off

bot

I'm just watching this thread with interest, but have no idea or opinion about why it may be occurring. I was just pointing out the reasoning that octoprint didn't rule out a faulty sd card.

kazolar

@bot I am using a brand new SanDisk 32 gb card. I have used 4 different cards. Are they all faulty?

And the fault magically clears after 40 minutes. And works fine for the next 15 hours...hell I had a night when I caught it in the morning right after it finished a print and was able to run another with no issues. Start cold and the SD card has a fault in it ... I'm software engineer and this makes no sense to me.

dc42

@kazolar said in Duet 2.05 memory leak?:

@dc42 what makes you think it's an SD card error still? I had the same underrun error reproduced with Ocotprint.

This thread is long, so it's hard to find your post in which you mentioned Octoprint (the browser search facility doesn't find it). I recall that you said you had experienced underruns with Octoprint, but AFAIR you didn't say you had experienced the same pattern of underruns i.e. the first print working and subsequent prints not. I replied that underruns when using Octoprint/USB were common on all electronics.

@kazolar said in Duet 2.05 memory leak?:

And the fault magically clears after 40 minutes.

40 minutes of what? The machine standing idle, or something else?

bot

One thought I randomly had last night: Power supplies often have programmed behaviour, based on times or temperatures. Perhaps your PSU is affecting the electronics when a fan kicks in, or something.

kazolar

@dc42 ok, so what's next then. It's warmed up and printing fine at present. I'll be able to get 2 more prints in today, but in the morning tomorrow it will require a failed print to clear the cobwebs again, get it warmed up to get going again...that feels like faulty hardware. Are we still searching for some fix. You tell me it's off warranty. I'll order a new on filastruder today.

kazolar

@bot fan on the main psu is hard wired with no temp sensor in the PSU, and runs all the time. The fan in the PSU runs off its own 12v regulator. The fans for cooling the duet and other electronics are 12v fans running of a separate fanless power supply. duet now, after fixing the z motor connectors is running at nice 26c. So clearly they were underspec'ed and running hot - enough to raise the board temp 3c.

kazolar

@dc42 if I start cold. Even if the machine has been powered for hours after finishing a print. Or off. Same things. The first print using the same procedure - heatup everything, wait for it come to temp, then hit reset start the print. This print fails with high loop times and eventual underruns, that takes 30-40 min. If I do the same procedure immediately following this print, subsequent prints work fine.

dc42

I have just re-read your original post, and spotted something I had forgotten:

@kazolar said in Duet 2.05 memory leak?:

I have done repeated duplication prints without issue -- but this leak happens during triplication -- so obviously more steppers are involved.

I can think of the following ways in which a fault might happen only when doing a triplicate print:

A power issue, if the PSU can't handle the additional heater and stepper motor load. What PSU are you using, and are the VIN terminal block screws still tight? However, your M122 report does not show any power outages, assuming you ran the M122 report just once at the end of the print because M122 resets the min and max voltages.
A temperature issue, because more stepper drivers and heaters are running. However, your M122 report shows a low MCU temperature, assuming you ran the M122 report just once at the end of the print because M122 resets the min and max temperatures.
A firmware issue that is causing DriveMovement objects to be lost from the system. However, your M122 report shows that there were never less than 56 of these free.
Noise on the bus between the Duet and DueX being worse when you drive another stepper motor on the DueX. This could cause I2C data corruption, or possibly spurious interrupts form the DueX to the Duet (which would lengthen the loop time).

None of these explain the increased loop time when this issue occurs. Whereas if SD card reads take longer, that would explain the increased loop time. Also, AFAIR you reported that re-uploading the print file cleared the problem - correct?

PS - it might be better to start a new thread about this issue (linking back to this one), starting by stating what you observe when the issue occurs (stuttering?), what precursors it has (increased underruns? increased loop time?), what conditions are needed to make it occur (triplicate not duplicate?), what has been proven to clear ot pre-empt the problem (M999? Power down/up? Upload the file again?) and what has been proven not to clear it.

bot

What specific PSU do you have?

Is it possible that it has some other behaviour? I know ATX supplies can switch things if no load is detected for a period of time. I am no expert by any means, but I just know that PSUs can behave how they decide is appropriate, and that is not always in line with the uses of them (if they are used in hobbyist projects and not for their exact intended purpose).

kazolar

@dc42 -- I have started a new thread:
https://forum.duet3d.com/topic/15862/duet-2-underruns-high-loop-times-stuttering

dc42

Thanks! I'll lock this one.