Duet 2.05 memory leak?

droftarts

=== DDARing ===
Scheduled moves: 1295584, completed moves: 1295544, StepErrors: 0, LaErrors: 0, Underruns: 595, 0

Thanks, I know where to look now!

Ian

kazolar

@droftarts I think I read that the first number is a warning, the 2nd number will cause stutter or a pause if it gets bad. I can tell from switching SD cards, my gcode uploads are faster now -- hitting 700kb/sec -- almost maxing out the 100mb link -- never had over 500 before.

bot

@kazolar I think the first value isn't a warning, just an indication that the lookahead function couldn't do something (not sure what) with the time given. It doesn't slow down the print, but is likely not ideal. The second number is a prepare move underrun, which means that the move could not be prepared in time and so the movement must wait. This is much worse than the first one.

Also, since I'm interested in SD card performance at the moment, I noticed your last comment and must correct you somewhat, just for your info: 700 kB/s is not nearly maxing out a 100 Mbps link. 100 Mbps = 12.5 MB/s

kazolar

@bot yep, 2nd number is the one that i would hope would be something Duet would alarm about -- yep -- I got my decimal off --was thinking 10mb, i think sleeping more than 5 hours per day maybe catching up to me. What's curious though is I never saw numbers above 500kb/s transfers -- with the new sd card it was in the high 700 -- touching 800, I mean even before I had issues with the SD card -- the old one wasn't that fast to begin with.

kazolar

@dc42 -- what kind of an sd card do i need:
it's happening again:
Scheduled moves: 24643, completed moves: 24607, StepErrors: 0, LaErrors: 0, Underruns: 80, 300

kazolar

@droftarts

10:42:27 AMSD write speed for 20.0Mbyte file was 2.18Mbytes/sec
10:42:18 AMM122 P104 S20
Testing SD card write speed...

I printed the same file multiple times -- no issues -- now I get an error -- does it have anything to do with duet being powered on/off. Works fine when I copy over a newly sliced/copied file -- but for a file that has been other for a couple of days - - start getting underruns again
Should i send a sd card dismount command m22 i think before powering the machine off -- something is definitely getting corrupted -- i did a bunch of commands, started, canceled a print -- and a newly copied file works fine.

kazolar

Pending any other surprises -- I now am sending M22 before the printer is powered off (unless it's a an unexpected shutdown) -- with that enabled, I have not had any more underrun problems.

kazolar

@dc42 totally reproducible now -- printed fine with the same file for 2 days, 3rd day after power up -- 30 minutes in underruns . Stopped it, didn't reset, didn't power off, deleted the file, uploaded the exact same file -- now no underruns, working fine. I guess I can keep doing this procedure, but why are "old" files now going stale on the sd card somehow?

Phaedrux

@kazolar And this is on 2.05? Or 2.02?

droftarts

@kazolar I can't see how anything in the firmware is doing this. It doesn't rewrite the file to the card, except if you run simulation (it appends the simulation time to the gcode file). My guess is that the SD card is doing some form of wear levelling, but causing issues doing it. What exact card (make and model) is this. I know you swapped to a new card from the one that was causing problems originally, but have you tried yet another?

Ian

kazolar

I had the problem originally with the card that came with the duet2, like 3 years ago, so I figured it must be time to swap. I had the same issue with amplim brand, then with sp brand. These have worked perfectly fine in pis. Both cards are rated class 10

kazolar

Also the problem is the same with each version. If I copy a fresh file, it works fine for a couple of days and 2 power cycles, then like clockwork the next print 30 minutes it starts to stutter, but I'm sure underruns were just piling up. I checked and the 2nd under run number was at 300, so it was pretty bad.

dc42

It certainly sounds like an issue with the SD card. Do you have logging enabled (M929)? That writes to the SD card at the start and end of each print, and when a significant event occurs. Writing to the log file could trigger the wear levelling process that Ian mentioned.

The only other possibility I can think of is a very obscure bug in the FatFs file system library.

kazolar

I don't have any specific logging enabled. Don't think anything writes to the sd card outside of a pause. Even issuing an m22 prior to shutdown didn't prevent it from happening. So the card was untouched, and the following power up... underruns and stuttering, then delete the gcode file upload it fresh and - perfect print, actually as I dial in the z offset, it's getting better each time...so the only file I update with some regularity it config.g

droftarts

@kazolar Is the soldering on the back of the SD socket okay? Maybe a pin is floating very occasionally. See https://duet3d.dozuki.com/Wiki/SD_Card#Section_SD_card_Socket for a poorly soldered one.

Ian

kazolar

@droftarts I'll check. Seems curious that it would be so predictable and odd failure. I have the duex5 under the duet in a case so it's gonna be hard to see. I was able to run off enough shields to get 150 out for today's donation drop to go to a neighboring towns health department, and we have more need in local nursing homes, so I'll be running it today to get more shields out. I'll probably just re-upload the file fresh to not tempt fate. Is it worthwhile to get like a brand name card, like SanDisk or something? The cards I got were in a 5 pack on Amazon awhile back, but we're very highly rated. Is the size a problem. I'm using 16gb cards.

droftarts

@kazolar It's a strange problem, indeed! Yes, up to 32GB is fine. See specification (and after for formatting details) here: https://duet3d.dozuki.com/Wiki/SD_Card#Section_Specification

I'd probably recommend getting name-brand (Kingston, SanDisk) cards, they're usually more reliable than no-name ones.

Ian

arhi

@kazolar said in Duet 2.05 memory leak?:

what kind of an sd card do i need:

Just try different card. I have new, original kingston's that would not at all work in rpi but work ok in opi, I have brand new original kingston's that behave weird in number of devices, I had issues with sandisk in duet2eth (not the one that died that was some PRC noname card, the sandisk I added later on) now with some Toshiba in one duet2eth and everything works ok and HAMA (the el cheapo 32G) and it all works ok. Not sure what's the deal with them, they all work properly in PC in card reader but in embedded systems some work and some don't. I stopped trying to find a "reason" and just change cards till I find one that works. So far never had issues with Toshiba's anywhere, most ugly is the RPI, it fails to work with number of cards, best of all is opi, it kinda works with anything you throw in it. Duet so far didn't like 2 different 4G Kingston and one 1G sandisk.

I didn't have stuttering btw, I had "garbage not valid G-code", and when you download the file from the duet it is garbage, and I had "weird moves" (middle of the print head moves to a weird position). Took me a while to suspect SD card first time but when the weird moves started SD card was the first step to check

One thing that might be taken from this, RPI is the wors embedded system I tested ever wrt SD cards so if a SD works on RPI it will most probably work everywhere. There is a list of tested SD cards with RPI https://elinux.org/RPi_SD_cards so if the card is there on the list and works ok with rpi it is usually a safe bet

kazolar

@droftarts so I did a fresh copy of the gcode file.
Running fine now...
I ordered a 32gb SanDisk card, will come tomorrow.
Some type of high performance 90mb/sec.
I think it's interesting that the duet thinks this card is perfectly fine:
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0

kazolar

@droftarts said in Duet 2.05 memory leak?:

M122 P104 S

ok, started having issues even on a clean copy of the file -- so time to swap cards again -- found a sandisk 8gb card in my bin of cards formatted fat32 64kb -- did a test -- performance is highest I've seen
SD write speed for 20.0Mbyte file was 2.72Mbytes/sec

I hope this one works today. Getting rather frustrating -- now I know what the problem is, but can't seem to find a more permanent solution -- almost want to get an microsd to emmc adapter -- seems these micosd cards are so flaky -- can't find a good one -- will check the slot for cold solder joints -- but i recall looking at it before and it was fine.