Duet 2.05 memory leak?
-
Also the problem is the same with each version. If I copy a fresh file, it works fine for a couple of days and 2 power cycles, then like clockwork the next print 30 minutes it starts to stutter, but I'm sure underruns were just piling up. I checked and the 2nd under run number was at 300, so it was pretty bad.
-
It certainly sounds like an issue with the SD card. Do you have logging enabled (M929)? That writes to the SD card at the start and end of each print, and when a significant event occurs. Writing to the log file could trigger the wear levelling process that Ian mentioned.
The only other possibility I can think of is a very obscure bug in the FatFs file system library.
-
I don't have any specific logging enabled. Don't think anything writes to the sd card outside of a pause. Even issuing an m22 prior to shutdown didn't prevent it from happening. So the card was untouched, and the following power up... underruns and stuttering, then delete the gcode file upload it fresh and - perfect print, actually as I dial in the z offset, it's getting better each time...so the only file I update with some regularity it config.g
-
@kazolar Is the soldering on the back of the SD socket okay? Maybe a pin is floating very occasionally. See https://duet3d.dozuki.com/Wiki/SD_Card#Section_SD_card_Socket for a poorly soldered one.
Ian
-
@droftarts I'll check. Seems curious that it would be so predictable and odd failure. I have the duex5 under the duet in a case so it's gonna be hard to see. I was able to run off enough shields to get 150 out for today's donation drop to go to a neighboring towns health department, and we have more need in local nursing homes, so I'll be running it today to get more shields out. I'll probably just re-upload the file fresh to not tempt fate. Is it worthwhile to get like a brand name card, like SanDisk or something? The cards I got were in a 5 pack on Amazon awhile back, but we're very highly rated. Is the size a problem. I'm using 16gb cards.
-
@kazolar It's a strange problem, indeed! Yes, up to 32GB is fine. See specification (and after for formatting details) here: https://duet3d.dozuki.com/Wiki/SD_Card#Section_Specification
I'd probably recommend getting name-brand (Kingston, SanDisk) cards, they're usually more reliable than no-name ones.
Ian
-
@kazolar said in Duet 2.05 memory leak?:
what kind of an sd card do i need:
Just try different card. I have new, original kingston's that would not at all work in rpi but work ok in opi, I have brand new original kingston's that behave weird in number of devices, I had issues with sandisk in duet2eth (not the one that died that was some PRC noname card, the sandisk I added later on) now with some Toshiba in one duet2eth and everything works ok and HAMA (the el cheapo 32G) and it all works ok. Not sure what's the deal with them, they all work properly in PC in card reader but in embedded systems some work and some don't. I stopped trying to find a "reason" and just change cards till I find one that works. So far never had issues with Toshiba's anywhere, most ugly is the RPI, it fails to work with number of cards, best of all is opi, it kinda works with anything you throw in it. Duet so far didn't like 2 different 4G Kingston and one 1G sandisk.
I didn't have stuttering btw, I had "garbage not valid G-code", and when you download the file from the duet it is garbage, and I had "weird moves" (middle of the print head moves to a weird position). Took me a while to suspect SD card first time but when the weird moves started SD card was the first step to check
One thing that might be taken from this, RPI is the wors embedded system I tested ever wrt SD cards so if a SD works on RPI it will most probably work everywhere. There is a list of tested SD cards with RPI https://elinux.org/RPi_SD_cards so if the card is there on the list and works ok with rpi it is usually a safe bet
-
@droftarts so I did a fresh copy of the gcode file.
Running fine now...
I ordered a 32gb SanDisk card, will come tomorrow.
Some type of high performance 90mb/sec.
I think it's interesting that the duet thinks this card is perfectly fine:
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0 -
@droftarts said in Duet 2.05 memory leak?:
M122 P104 S
ok, started having issues even on a clean copy of the file -- so time to swap cards again -- found a sandisk 8gb card in my bin of cards formatted fat32 64kb -- did a test -- performance is highest I've seen
SD write speed for 20.0Mbyte file was 2.72Mbytes/secI hope this one works today. Getting rather frustrating -- now I know what the problem is, but can't seem to find a more permanent solution -- almost want to get an microsd to emmc adapter -- seems these micosd cards are so flaky -- can't find a good one -- will check the slot for cold solder joints -- but i recall looking at it before and it was fine.
-
A few of our OEMs are using branded industrial-grade SDHC cards in Duets instead of consumer-grade ones. They have a much greater write endurance, but they cost a lot more. Perhaps the consumer-grade ones are not so great for read endurance either. They are available from the usual electronic component distributors (Digikey, Farnell, RS, Mouser, Newark etc.).
-
You can get "high endurance" cards on amazon as well for not much more.
I doubt those cards are actually using SLC flash though like the true industrial ones are using. SLC flash is much more expensive because the capacities are limited due to real estate limits.
300-400$ for a 16gb UHC1 SLC card.
More info on the difference between flash technologies: https://www.mydigitaldiscount.com/everything-you-need-to-know-about-slc-mlc-and-tlc-nand-flash.html
-
@kazolar said in Duet 2.05 memory leak?:
ok, started having issues even on a clean copy of the file
Are you 100% sure your sd-card slot is soldered properly? I think there was already a question but I did not see the answer.
-
@arhi i will check again after this print finishes -- but I got a sandisk 8gb card in there now, and no issues -- printing happily without a single underrun error.
-
I haven't ran m122 for couple of hours -- so this is a decent sample
Slowest loop: 8.62ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 144, MinFreeDm: 6, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 196268, completed moves: 196228, StepErrors: 0, LaErrors: 0, Underruns: 399, 0I mean -- slowest loop is under 10ms -- can't see how anything can be wrong -- and the important underrun number is 0.
-
@kazolar lot of guessing now but as someone who did have a lot of SD card issues on embedded systems what you are writing does not ring a bell. SD cards do behave weird but "freshly recorded" is not something I experienced ever, on any system. Reboot the system and SD start working is even weirder. The way you are describing the problem, to me, more looks like the temperature of the board and thickness of the sd-card change the contact between PCB and SD-card slot. So a cold joint of a kind there. Those card slots are often improperly soldered, and sometimes they can appear ok but a hairline fracture can exist end temperature/vibration can disconnect it. It's bin a long time since I used fatfs library but IIRC there is the checksum for reading data so this should be detected, the question is how RRF handles the retries as many retries might be seen as those stutters and underruns. Maybe run a SD card test on the duet and add physical stress to the board while it's running (slight bend, twist...)
-
@kazolar said in Duet 2.05 memory leak?:
I haven't ran m122 for couple of hours -- so this is a decent sample
Slowest loop: 8.62ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 144, MinFreeDm: 6, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 196268, completed moves: 196228, StepErrors: 0, LaErrors: 0, Underruns: 399, 0I mean -- slowest loop is under 10ms -- can't see how anything can be wrong -- and the important underrun number is 0.
What about the max SD retries?
-
@dc42
today with the sandisk card it was
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0I did what others have asked and took another look at the microSD card solder joints again -- and saw nothing suspicious -- and short of halting everything to pull the board out and stick under my scope and touch up the solder joints -- this doesn't look faulty.
Here is a full res picture
https://www.dropbox.com/s/197mtwcmt2vq6co/SDCard.jpg?dl=0 -
so more underruns -- getting to a point where I think I need to format a card clean to get it to do an error free print..At this point I am pretty sure it's not the card. I tried shaking the enclosure and so on while running a write test and I am getting 3.03 and 3.14mb/sec -- even faster than it was before. I am trying again with a clean upload of a file. The problem seems to have progressively gotten worse -- before I could at least reset the board and print without issues, now that doesn't even work.
-
that slot looks ok, if the stuttering was due to the sd card I assume there would be max retries there
-
Fresh copy -- no reset
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 27.4, current 27.6, max 28.1
Supply voltage: min 24.1, current 24.4, max 24.6, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: ok, SG min/max 0/341
Driver 1: standstill, SG min/max not available
Driver 2: ok, SG min/max 0/1023
Driver 3: ok, SG min/max 0/310
Driver 4: ok, SG min/max 0/321
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max 51/258
Driver 7: standstill, SG min/max 159/297
Driver 8: standstill, SG min/max 0/208
Driver 9: standstill, SG min/max 0/221
Date/time: 2020-04-20 22:01:22
Cache data hit count 4294967295
Slowest loop: 6.01ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 128, MinFreeDm: 6, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 26040, completed moves: 26000, StepErrors: 0, LaErrors: 0, Underruns: 70, 036 minutes in -- previous attempt that failed was after examining the SD card. Slowest loop was bad, underruns were piling up -- I didn't notice what the write retries count was -- but I think it was zero, since it's not writing, but reading
