Duet 2.05 memory leak?
-
@dc42 totally reproducible now -- printed fine with the same file for 2 days, 3rd day after power up -- 30 minutes in underruns . Stopped it, didn't reset, didn't power off, deleted the file, uploaded the exact same file -- now no underruns, working fine. I guess I can keep doing this procedure, but why are "old" files now going stale on the sd card somehow?
-
@kazolar And this is on 2.05? Or 2.02?
-
@kazolar I can't see how anything in the firmware is doing this. It doesn't rewrite the file to the card, except if you run simulation (it appends the simulation time to the gcode file). My guess is that the SD card is doing some form of wear levelling, but causing issues doing it. What exact card (make and model) is this. I know you swapped to a new card from the one that was causing problems originally, but have you tried yet another?
Ian
-
I had the problem originally with the card that came with the duet2, like 3 years ago, so I figured it must be time to swap. I had the same issue with amplim brand, then with sp brand. These have worked perfectly fine in pis. Both cards are rated class 10
-
Also the problem is the same with each version. If I copy a fresh file, it works fine for a couple of days and 2 power cycles, then like clockwork the next print 30 minutes it starts to stutter, but I'm sure underruns were just piling up. I checked and the 2nd under run number was at 300, so it was pretty bad.
-
It certainly sounds like an issue with the SD card. Do you have logging enabled (M929)? That writes to the SD card at the start and end of each print, and when a significant event occurs. Writing to the log file could trigger the wear levelling process that Ian mentioned.
The only other possibility I can think of is a very obscure bug in the FatFs file system library.
-
I don't have any specific logging enabled. Don't think anything writes to the sd card outside of a pause. Even issuing an m22 prior to shutdown didn't prevent it from happening. So the card was untouched, and the following power up... underruns and stuttering, then delete the gcode file upload it fresh and - perfect print, actually as I dial in the z offset, it's getting better each time...so the only file I update with some regularity it config.g
-
@kazolar Is the soldering on the back of the SD socket okay? Maybe a pin is floating very occasionally. See https://duet3d.dozuki.com/Wiki/SD_Card#Section_SD_card_Socket for a poorly soldered one.
Ian
Bed-slinger - Mini5+ WiFi/1LC | RRP Fisher v1 - D2 WiFi | Polargraph - D2 WiFi | TronXY X5S - 6HC/Roto | CNC router - 6HC | Tractus3D T1250 - D2 Eth
-
@droftarts I'll check. Seems curious that it would be so predictable and odd failure. I have the duex5 under the duet in a case so it's gonna be hard to see. I was able to run off enough shields to get 150 out for today's donation drop to go to a neighboring towns health department, and we have more need in local nursing homes, so I'll be running it today to get more shields out. I'll probably just re-upload the file fresh to not tempt fate. Is it worthwhile to get like a brand name card, like SanDisk or something? The cards I got were in a 5 pack on Amazon awhile back, but we're very highly rated. Is the size a problem. I'm using 16gb cards.
-
@kazolar It's a strange problem, indeed! Yes, up to 32GB is fine. See specification (and after for formatting details) here: https://duet3d.dozuki.com/Wiki/SD_Card#Section_Specification
I'd probably recommend getting name-brand (Kingston, SanDisk) cards, they're usually more reliable than no-name ones.
Ian
-
@kazolar said in Duet 2.05 memory leak?:
what kind of an sd card do i need:
Just try different card. I have new, original kingston's that would not at all work in rpi but work ok in opi, I have brand new original kingston's that behave weird in number of devices, I had issues with sandisk in duet2eth (not the one that died that was some PRC noname card, the sandisk I added later on) now with some Toshiba in one duet2eth and everything works ok and HAMA (the el cheapo 32G) and it all works ok. Not sure what's the deal with them, they all work properly in PC in card reader but in embedded systems some work and some don't. I stopped trying to find a "reason" and just change cards till I find one that works. So far never had issues with Toshiba's anywhere, most ugly is the RPI, it fails to work with number of cards, best of all is opi, it kinda works with anything you throw in it. Duet so far didn't like 2 different 4G Kingston and one 1G sandisk.
I didn't have stuttering btw, I had "garbage not valid G-code", and when you download the file from the duet it is garbage, and I had "weird moves" (middle of the print head moves to a weird position). Took me a while to suspect SD card first time but when the weird moves started SD card was the first step to check
One thing that might be taken from this, RPI is the wors embedded system I tested ever wrt SD cards so if a SD works on RPI it will most probably work everywhere. There is a list of tested SD cards with RPI https://elinux.org/RPi_SD_cards so if the card is there on the list and works ok with rpi it is usually a safe bet
-
@droftarts so I did a fresh copy of the gcode file.
Running fine now...
I ordered a 32gb SanDisk card, will come tomorrow.
Some type of high performance 90mb/sec.
I think it's interesting that the duet thinks this card is perfectly fine:
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0 -
@droftarts said in Duet 2.05 memory leak?:
M122 P104 S
ok, started having issues even on a clean copy of the file -- so time to swap cards again -- found a sandisk 8gb card in my bin of cards formatted fat32 64kb -- did a test -- performance is highest I've seen
SD write speed for 20.0Mbyte file was 2.72Mbytes/secI hope this one works today. Getting rather frustrating -- now I know what the problem is, but can't seem to find a more permanent solution -- almost want to get an microsd to emmc adapter -- seems these micosd cards are so flaky -- can't find a good one -- will check the slot for cold solder joints -- but i recall looking at it before and it was fine.
-
A few of our OEMs are using branded industrial-grade SDHC cards in Duets instead of consumer-grade ones. They have a much greater write endurance, but they cost a lot more. Perhaps the consumer-grade ones are not so great for read endurance either. They are available from the usual electronic component distributors (Digikey, Farnell, RS, Mouser, Newark etc.).
-
You can get "high endurance" cards on amazon as well for not much more.
I doubt those cards are actually using SLC flash though like the true industrial ones are using. SLC flash is much more expensive because the capacities are limited due to real estate limits.
300-400$ for a 16gb UHC1 SLC card.
More info on the difference between flash technologies: https://www.mydigitaldiscount.com/everything-you-need-to-know-about-slc-mlc-and-tlc-nand-flash.html
-
@kazolar said in Duet 2.05 memory leak?:
ok, started having issues even on a clean copy of the file
Are you 100% sure your sd-card slot is soldered properly? I think there was already a question but I did not see the answer.
-
@arhi i will check again after this print finishes -- but I got a sandisk 8gb card in there now, and no issues -- printing happily without a single underrun error.
-
I haven't ran m122 for couple of hours -- so this is a decent sample
Slowest loop: 8.62ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 144, MinFreeDm: 6, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 196268, completed moves: 196228, StepErrors: 0, LaErrors: 0, Underruns: 399, 0I mean -- slowest loop is under 10ms -- can't see how anything can be wrong -- and the important underrun number is 0.
-
@kazolar lot of guessing now but as someone who did have a lot of SD card issues on embedded systems what you are writing does not ring a bell. SD cards do behave weird but "freshly recorded" is not something I experienced ever, on any system. Reboot the system and SD start working is even weirder. The way you are describing the problem, to me, more looks like the temperature of the board and thickness of the sd-card change the contact between PCB and SD-card slot. So a cold joint of a kind there. Those card slots are often improperly soldered, and sometimes they can appear ok but a hairline fracture can exist end temperature/vibration can disconnect it. It's bin a long time since I used fatfs library but IIRC there is the checksum for reading data so this should be detected, the question is how RRF handles the retries as many retries might be seen as those stutters and underruns. Maybe run a SD card test on the duet and add physical stress to the board while it's running (slight bend, twist...)
-
@kazolar said in Duet 2.05 memory leak?:
I haven't ran m122 for couple of hours -- so this is a decent sample
Slowest loop: 8.62ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 144, MinFreeDm: 6, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 196268, completed moves: 196228, StepErrors: 0, LaErrors: 0, Underruns: 399, 0I mean -- slowest loop is under 10ms -- can't see how anything can be wrong -- and the important underrun number is 0.
What about the max SD retries?
