Duet 2.05 memory leak?
-
@Phaedrux so I'm a 3rd print in with the sandisk 8gb card (with 32kb cluster size) -- and no issues -- at least I'm past the point when I would normally run into an issue -- i got a new 32gb sandisk card delivered today, but if this keeps working -- I'm not gun ho to swap it.
If this survives a power down later and power up and start print of the same file -- then I'm keeping the setup as is and holding on to the 32 gb card for another time.As far as triplicate printing vs single/double -- difference is that more hotends and steppers are involved, and both Y gantries are moving -- presumably quad would involve the 4th x axis as well. I've never done that since my 4th extruder has a smaller nozzle -- and it's dedicated for detail features. I am able to get really good yields with triplicate printing, so I haven't had the need to setup the 4th one with the same size as the the other 3.
I just find it odd that triplicate print is triggering this issue and the duplicate print of 26 ear relievers -- so 52 in total worked flawlessly twice. And that gcode file was bigger than the shield gcode file I'm running now. -
The 8 gig card lasted 3 prints, 4th -- failed -- I'm beginning to suspect the microsd slot or related circuitry -- I took the new card -- left it with whatever formatting it came with -- and not right away like before at the 30 minute mark, but about 40 minute, I got underruns. I then reformatted it with 64kb cluster size. Pressed firmly on the casing of the microsd card, and reseted the ethernet header -- got everything tightened up, and -- it's printing fine. Should I bother trying to take the board out and reflow the microsd card. It's far from trivial to take the board out with so much connected to it. I was thinking of upgrading to duet 3 when things were less hectic, but I need this to work just while the PPE shortage is still high. I got this board from filastruder and nothing had been wrong with it -- I'd own up to it if I did something -- cause the first duet 2 that was part of this printer -- that was on me -- I didn't come here saying -- oh my board is not working I don't know what I did. I had a short and it killed it -- this is working fine, then not. Feels wasteful to go on buying another one cause I'd rather take my time get a duet 3 get several expansion modules and rewire the printer.
-
You've said that re-uploading the file after 3 prints clears the problem. So i don't think it is a problem with the Duet hardware.
Underruns in themselves do not necessarily indicate a problem - you may get some anyway if parts of your print contain long runs of very short segment. However, if SD card read operations become slow for any reason, you will get increased underruns, and eventually stuttering.
Please confirm (I think you have already said it) that resetting the Duet doesn't clear the problem, neither does powering the Duet down (including removing USB or 5V external power, if you are using it).
Regarding cluster size, we recommend using the largest one you can, which is normally 64kb. That reduces the number of SD card accesses needed.
-
@dc42 I don't have external 5v or USB connected. What I did last night to get it going again was -- I powered it down, reformatted the card -- and copied all the backed up data to it -- and it printed fine overnight without ANY underruns (i'm only looking at the last number) when it starts getting underruns in the 2nd number, it starts stuttering very quickly, and the underrun numbers keep going up, I haven't seen it not stutter and the 2nd number be >0. It seems that a fresh format or a fresh copy clears up whatever is wrong -- my guess in terms of hardware is maybe like others said 3 prints shook the machine, the difference with triplication is that 2 gantries are involved, and a lot more movement, so whatever wouldn't vibrate the machine on a single or a duplication print, would vibrate it on triplication -- then me taking the card out and re-seating it -- and starting a new print, makes better contacts. I had previously been successful just doing a reset and starting the print on a fresh boot - but that stopped working at some point -- or I stopped trying after 2.5.1 update. I can try to see if I can get more than 3 prints in a row by doing resets between each
-
What slicer are you using to generate the gcode? Can you post a sample?
-
Here is the file
https://www.dropbox.com/s/6myicwelo20mzjd/VisorQUADQuad.gcode?dl=0I just tried printing it -- and there was stuttering and underruns within 20 minutes -- this was after the board was reset (not powered down) using the file that worked fine for the overnight print. I just did a fresh copy of the file. I started the print again -- I saw 22 underruns in the 2nd number after a couple of minutes, but no noticeable stuttering. Not yet. I will keep checking until about an hour in, if it prints stutter free for 45+ min plus, then it will be fine through the course of the 5 hour print -- stuttering due to underruns starts in the first 30-40 minutes if it happens. Never seen it start later than that.
Looks like this print with a fresh copy is working fine (first 22 underruns appear to be innocuous)
-
I know it's not trivial, but is it possible to switch to a slicer other than S3D?
-
@Phaedrux no other slice supports different diameter nozzles for different extruders. I tried setting up in cura recently and that wasn't an option. My startup script uses s3d specific variables for temperature setting. Other slicers are better than s3d on many respects, but s3d is still better at multi extruder setup -- since it can all be done via multiple processes -- so it's still not an option for me.
-
@Phaedrux -- yep, running fine now with a fresh copy. Underruns are not piling up doing fine -- I can check S3D option for decreasing small movements -- not sure if that has anything with what's going on.
-
I suspected as much. S3D can create some odd gcode with lots of small segments sometimes which may be half the problem.
I wasn't aware of the nozzle size limitation in Cura. I was pretty sure it was an option the last time I had multiple extruders setup for the Palette 2.
-
@Phaedrux Palette 2 uses 1 nozzle -- I have palette+. It's effectively a mixing hotend, same prusa slicer -- it's geared for the MMU -- which is still a single nozzle. So there is no reason to specify different sizes for extruders. This has been a limitation since I first tried to use multi extrusion on my maker gear i had a 0.35 nozzle and 0.5 -- and I could only set it up in S3D. There is only a python script that I could that would slim down S3D gcode to avoid unnecessary moves. However -- why is it printing fine and happy now underrun free -- nearly an hour in -- I know this print will succeed. The pain point is 30 minutes in. What's odd to me is the cura is developed by ultimaker and they have the UM3 with their different print cores, and I would think having the ability to use the 2 print cores for different resolution would be something they'd want to be able to do, but I guess they expect you to match print core nozzle diameters.
-
@kazolar said in Duet 2.05 memory leak?:
@Phaedrux Palette 2 uses 1 nozzle -- I have palette+. It's effectively a mixing hotend, same prusa slicer -- it's geared for the MMU -- which is still a single nozzle. So there is no reason to specify different sizes for extruders. This has been a limitation since I first tried to use multi extrusion on my maker gear i had a 0.35 nozzle and 0.5 -- and I could only set it up in S3D. There is only a python script that I could that would slim down S3D gcode to avoid unnecessary moves. However -- why is it printing fine and happy now underrun free -- nearly an hour in -- I know this print will succeed. The pain point is 30 minutes in. What's odd to me is the cura is developed by ultimaker and they have the UM3 with their different print cores, and I would think having the ability to use the 2 print cores for different resolution would be something they'd want to be able to do, but I guess they expect you to match print core nozzle diameters.
Cura should have no issue setting different nozzle diameters, unless I'm completely missing something with your setup. In Cura you can easily set the nozzle diameter for different extruders, and then from there also modifiy extrusion widths.
As an example an IDEX machine with two different nozzles (.4mm and 1mm)
A warning, the two images below aren't a perfect example because of the profile I have set, but it shows what I mean - not only different nozzle diameters configured, but different line widths.
Regarding S3D Processes (I have not used S3D but read about how they work... I think) - you can do similar model manipulations (depending on your job) by changing what parts of a model are made by what extruder, e.g. outer walls with Extruder #1, infill with Extrudr #2, support with Extruder #3, just by using the standard settings. You can also do more advanced modifications with the "Per Model Settings" feature ("Normal Model", "Print as Support", "Modifiy settings for overlaps", "Don't support overlaps").
-
@sebkritikel thanks, will look at it again. It must be a somewhat recent addition. I couldn't do this before -- I'll need to look at it again. Though it seems odd that depending on the time of day random conditions -- this print works or doesn't . Btw -- I am running THE EXACT same print on my modified series 1 (1 extruder) -- running a smoothieboard (azteeg something) -- controlled by octoprint and it runs perfectly fine.
I'm almost thinking I should just ditch duet's native gcode feeder and switch to octoprint. Kinda makes the screen from duet useless. -
@Phaedrux I really don't want to turn this into an S3D vs Cura debate. That never goes well. I've ran this print with no issues 3 times yesterday -- then the 4th had underruns. If it's a hadware fault -- I can swap the duet 2 -- I don't have the time to setup duet 3 now with swapping expansion boards and so on. If this isn't a firmware bug, that's the only answer left. I've gone through 4 SD cards. I'm using a branded sd card now. I don't know what else to do.
-
@kazolar said in Duet 2.05 memory leak?:
S3D vs Cura debate
Not the intention. Just trying to chase things down.
-
@Phaedrux I have approval from my employer (they're sponsoring the costs of producing PPE) to swap the board if that what needs to happen, or go the duet 3 route (they also agreed to cover the cost of that), but that rewire job is not something I am comfortable doing now since it will put the machine out of commission while we still have a significant need for PPE. I've personally produced 810 complete shields -- and our group of 3 printers (people) + 1 logistics person has delivered over 1600 shields together, as you can see I am responsible for more than 50% of our production -- because of the quad (QuadZilla) being able to produce shields in large batches.
If you guys tell me OK -- the board is off warranty we can't offer an RMA, and getting new one will fix it -- I'll place the order to filastruder today -- and get it before the end of the week, and 1 hour rewire I'll be up and running. I'm working from home so going to the basement every few hours to cycle prints isn't critical, but dealing with underrun errors takes more time than i should really be putting in during the day. -
Well we'd certainly like to get to the bottom of this one, and unfortunately that would probably require some more disruptive troubleshooting than would be ideal given the situation.
Please hold off for the moment while we see what we can do.
-
@kazolar I have just skimmed the thread - thanks for the persistence here, can I attempt to summarise to be clear where we are at:
- This problem occurs with 2.05.1 (with your modifications) but not with 2.03 RC2?
- When it happens its repeatable (i.e. it happens at the same point (after 30 minutes) but not every time, can you still provoke it by resetting, or not resetting the board, or any other action?
- Are you still seeing large numbers of underruns when the problem occurs? Maybe check underruns at the same point in time after the start of a print (30 minutes?) to see if there is a correlation.
- You are not able to test this with gcode generated by another slicer? and it only occurs when printing in triplicate mode?
-
@T3P3Tony
1 - it's reproducible with 2.03 RC 2 as well. Curiously enough I did a bunch of triplicate prints in the summer with this version with no issues, so I tried it -- but problem happens there as well.
2 - semi reproducible -- seems that if I keep printing the same file, and the 3rd or 4th print of the same file will trigger the issue. Resetting the board doesn't help. Seems making a fresh copy of the same exact file -- or formatting card (even without resetting or powering the board down, formatting requires a power down) -- gets it to a working state (as it's now)
3. When the problem occurs and I see stuttering -- underruns go into triple digits very quickly, last night 4th print -- at 27 minute mark it was fine, I thought I was in the clear. I was watching a movie -- I checked my phone 10 minutes later underruns were at 200+ It's technically exactly 30 minutes after the lead screw compensation finishes -- it may take a couple of minutes from the moment I hit start to when all temps stabilize and it will do the probing routine. So it's technically closer to 32 --33 minutes, but the 30 minute mark is right after the screen displays the result of lead screw compensation.
4. Setting up this print with another slicer is going to take some time. I have very specific settings in S3D which are tuned to produce these shields as fast as possible. Triplicate mode differs from duplicate primarily that the 2nd Y gantry (technically U) is involved, and printer vibrates more -- it's 200lb machine, but the gantries are more akin to CNC gantries so -- there is a lot of force involved.
I've not had it happen in duplicate mode, but I've only ran duplicate prints a couple of times, it seems to take 3-4 prints to basically render the file unprintable.
5. Whatever it's worth -- with 2.05 -- If I did anything (that involves movement) -- even try to home the machine prior to starting this triplicate print -- I'd get this behavior -- 30 min after lead screw compensation it would start getting underruns and my approach was to simply reset the board -- pre-heat, clean the nozzles -- I need to clean at least tool 0 -- I use piezo to probe. Then I'd start the print and that worked for a while..then even that stopped working and only thing that would resolve things is clean copy or card reformat.
6. Yesterday I had the most success -- 3 error free prints in a row after I formatted an 8 gb sandisk card with 32kb cluster size. After the failure i switched to the new 32gb sandisk card I got yesterday formatted with 64kb cluster size. The first print with that card was fine, 2nd I tried this morning -- didn't work. I deleted the -- uploaded a new copy -- it's running fine 3 hrs 36min -- no underruns worth noting (22 at the beginning, but no ill effect from them), 0 since. -
@kazolar you are not running stock firmware right? you are patching it and compiling yourself?
running few prints using octoprint instead from duet sd card would imo for sure completely remove sd card from the picture (or prove it has to do with sdcard/sdcard socket) so that might be a good test.
as for the processes and extruders ideamaker started as a copy of s3d so has most stuff s3d has + some stuff extra, you might want to test it out. it is not open source but it is free
