Duet 2.05 memory leak?
-
@kazolar said in Duet 2.05 memory leak?:
m18 -- so then we're no longer at a known state. Homing z gets all the lead screws close -- my Z compensation adjustments are < 0.5mm -- I adjusted those end stops to be as close to reality as possible.
If you can remove the M18 from the ends gcode so there is no need to rehome and then see - It would be good if we can tie this down to running that Z levelling, or not.
-
@Phaedrux I actually am not a big fan of the new web UI so I'm using reprap.htm -- and that goes to 1.22.6
The DWC that is loaded is Duet Web Control 2.0.0-RC2, not to be a critic, but the new look is not my thing, i liked the old one better -- glad to still be able to use it.
@T3P3Tony
The problem is my startup gcode does that too, it goes through a cycle of putting every tool into standby -- I'd have to be very careful with what I change in these sequences. The combination tfree5.g tpost0.g -- has some odd consequences. I am not sure what's going on -- I haven't sorted that gcode out -- so I could try and cut out all but the essentials in start and stop gcode -- obviously that I can only do during the day when I'm here to observe. My plan for today is -- if it underruns again -- print is running now only a few minutes in -- I will setup octoprint and feed gcode that way -- if things work via octoprint, then it's pretty clear the fault is in the sdcard circuitry. -
@T3P3Tony by the way lead screw compensation result -- probing 5 points, center, and near each lead screw is 0.004 in the print that just started seen at all zeros before -- gotta love the single start lead screws and a milled aluminum plate.
-
@kazolar said in Duet 2.05 memory leak?:
I actually am not a big fan of the new web UI
Fair enough, but if it's a CRC issue that's corrupting the files it would be good to test with a known working DWC version. If you update to 2.0.7 you can still use the 1.x version as before.
-
@Phaedrux will do after this print either fails or succeeds -- about 10 minutes left to find out
-
40 minutes into the 2nd print of the day -- had to change 2 filament spools -- this thing eats filament like you wouldn't believe. No underruns (if one was going to happen, it would have happened already). I actually read the lead screw compensation result from an earlier print -- this one said 0.001 -- I'd say the bed is flat.
-
@Phaedrux so a late night for me -- very late night, we;ll, actually didn't go to bed. But I got some interesting results.
I did as you asked -- I updated my DWC went with 2.1-- and the very next print -- I got underruns -- I then went ahead and figured I'd bypass the SD card and run from octoprint -- I thought that would solve it -- nope, underruns again, stuttering the full boat -- so I figured I'd look at my firmware changes a bit -- and I removed a few of them to bring things back to more stock -- left just the minimum I need for the config. Again -- underruns -- stuttering and it just ends up freezing and rebooting - not right away, but 30 minutes into the print as usual, -basically crashing with a m122 showing the slowest loop being at over 4 seconds, by then it's complaining about drivers, earth, moon, and the stars. So I figured to return my firmware back to how I had it before -- as it did work several times during the day. Tried again -- underruns.
Next -- I gave a shot with formatting the SD card again -- tried 32kb cluster size, just to see what would happen -- this card doesn't work right at that size -- it's a 32gb card, so that was glitchy from the start -- all gcodes were executing slow -- so I formatted again to 64kb -- and -- long story short -- I'm back to 2.05RC -- not 2.05.1 -- I need to build that again and figure out what I reversed and shouldn't have -- but I broke one of my changes -- and stupidly I didn't back up the binary that was working today -- so I'm back at a firmware that did work (with resets) -- and is working now. I am far enough into the print that I know it will be fine...so here is the interesting part.
During the first 30 minutes when I initially thought it could go either way, well now doing M122 a few times, I know that's not the case. I see that if slowest loop start inching up towards 100ms -- it will eventually fail and start to underrun -- the issue is not the sd card, it's the fact that loop gets bogged down, with something -- now that it's fine slowest loop is 6-10ms -- so hence no underruns. So blaming the SD card was a mistake -- I wish I hadn't pried up the posts on the ethernet adapter to look at the sd card solder joints -- because that's not fully secure anymore.
So it's pretty obvious that the sd card is not at fault -- underruns are happening because the loop gets slow -- why -- that's not something I have clue about, but I can see that DWC upgrade made things worse. I'm almost tempted to go back to the 1.22.6 DWC and remove 2.0x whatever -- it looks like as soon as updated the DWC -- it literally took me all night to get it to start to print successfully. BTW -- I am back to my backed up version of older DWC -- 2.0.0RC2 -- the combination appears to work better. -
@kazolar said in Duet 2.05 memory leak?:
I can see that DWC upgrade made things worse
Not sure this is connected as DWC should not be influencing the loop time.
I think its time to bite the bullet and try a different slicer!
-
@T3P3Tony I started setting up Cura. It's taking some time to get it setup with the extruders -- I kinda remember why I paid for S3D, cura is slow now as it was 5 years ago. I can see it's a more complex geometry -- I'll try to re-slice it in Cura for next print and see where it goes
-
@T3P3Tony so just moving the basics into cura -- it's estimating the time spot on -- but it's generating a gcode that is 50% bigger -- probably that will cause more problems than fix them? I could try it, but my thinking was to eliminate small movements -- well I adjusted the mesh in Fusion 360 also to be less dense, and cut the gcode down a bit - but it still hit underrun errors. I am not sure the point in trying a different slicer if the gcode I am getting is bigger -- either it will do a lot more travels, or create even more small movements
-
@kazolar, when the problem occurs, have you tried uploading a different file to SD card, and then printing the original file again? I'm wondering whether it is the act of writing to the SD card that clears the problem.
Also, are you running with logging (M929) enabled or disabled? Logging will generate occasional writes to the SD card, e.g. when a print starts or finishes.
-
@dc42 i've been trying different things all night -- doesn't seem to be sd card specific. I tried printing from octoprint -- bypassing the SD card, and still hit underruns. I don't have M929 enabled. It seems that a full reset does clear the cobwebs -- I kept trying to preheat then clean the nozzle -- then print, but the print that is working fine -- 2+ hours into it now, I preheated, cleaned the nozzle, then hit reset -- then print -- that worked.
-
@kazolar said in Duet 2.05 memory leak?:
It seems that a full reset does clear the cobwebs
Is that with or without powering down and up again?
@kazolar said in Duet 2.05 memory leak?:
I tried printing from octoprint -- bypassing the SD card, and still hit underruns.
Underruns are likely when printing over USB if the GCode file contains sequences of short segments, unless the sending program knows that when talking to a Duet, it doesn't need to wait for an OK reply to each command. I've seen many reports (not from Duet users) that printing from Octoprint is sometimes slower than printing from SD card. So results using Octoprint are only significant if you compare multiple runs using Octoprint.
-
@dc42 i was able to get it going on again by simply hitting reset -- no power down, and immediately select a file to print. That's how it's printing now. It seems that if I do anything prior to initiating the print -- it starts to show significant max times in loop and then at the 30 minute mark it can't keep up and it starts getting underruns. I tweaked octoprint settings not to wait for OK before I started because it was timing out on basic, stuff -- it still hit underrun at about the same point. And I could see in DWC console when I did m122 -- I'd see loop times in the 50ms -- so it clearly wasn't going well. I check the model and the file for small segments, and there aren't any. I took that set of models into cura with adjustments for minimum travels, and it generated a bigger gcode file than S3D -- I'll try prusa slicer -- not sure what else to try. The problem has gotten progressively worse. The gcode file I'm printing 12 mb, the cura generated one with adjustments for minimum segments is 19mb -- I can print that file next and see what happens?
-
The different size could be Cura being more verbose with comments so its still worth trying.
-
@T3P3Tony I did a Prusa Slicer version -- it's a little bit bigger, but the paths look good -- I'm trying it now -- the previous print just finished -- I only added my custom header and tail to the prusa version to make sure the right stuff happens -- did not reset the board -- just uploaded the prusa version and see what happens.
-
@T3P3Tony Prusa version was doing funny things with perimeters -- sorta failed on it's own, kinda chewed up the perimeters of the model, I canceled it -- I'd need to mess with it to make it work -- Cura is next.
-
@T3P3Tony ok so the cura version -- 3rd print without a reset - right as it got to post5.g -- it was doing it in slow motion -- doing a move -- thinking doing another -- move. So I reset it -- will try again
-
@T3P3Tony ok -- so cura sliced code seemed a bit more sane than Prusa, but it didn't look good, but I let it run -- and it just failed -- underruns and stuttering.
I optimized it for short paths to simplify it -- still...same result. Gonna clean the bed. Reset and kick of the S3D print -- that actually looks good and works after a reset (fingers crossed) -
@T3P3Tony @dc42
Here is where I am at --- I am running 2.05RC -- 2.05.1 is a lot less predictable -- it seems to work ok, then stops working and gets into stuttering underrun condition. I can't trust it from print to print, so I went back to 2.05RC -- it's at repeatable, and I can get shields out now without checking on it every 5 minutes.
- I can tell right away from m122 after the print starts, actually starts laying down plastic -- if I run M122 two times in 30 second intervals, if the 2nd time I get
Slowest loop: 3.99ms; fastest: 0.08ms
Then it will work through the end of the print -- I can tell that within 5 minutes now of starting the print before waiting the full 30 minutes to get underruns ans stuttering .
If I get
Slowest loop: 63.64ms; fastest: 0.07ms
Then it will underrun by 30 minute mark. - This is what works. I preheated the bed, nozzles -- clean everything off - press reset -- then start the S3D print -- and this works. The print will get slowest loop in 5ms range and will run through to conclusion - 5hrs 15 min give or take.
- This procedure works -- and has worked for the entire month I've been printing shields until I started messing around with 2.05.1. I don't have to copy the file over, I can keep using the same file.
- If I do anything prior to starting the print without pressing reset -- I will get the high slowest loop numbers and inevitable underrun.
- The behavior is absolutely repeatable with S3D or Cura prints (in terms of underrun condition). I am sure I can tune the Cura version to look better, and probably the Prusa version, both looked like trash on layer one to varying degrees -- but both were optimized to eliminate small movements, S3D doesn't have that feature, but this print doesn't have any -- or many.
Question is -- is this hardware or software. I have done triplicate prints before without having to do the "three finger salute" before starting it -- so either something died in the hardware, David's documentation on underrruns talks about voltage regulators, etc. I have no indication that something is wrong there, but the only alteration from then and now, is I switched to optical end stops on XYUVWA axes - the others (for the bed -- are still mechanical). I also went from 32 teeth pulleys to 20 teeth. I have done a 40 hour long multi extrusion print with the current setup with no issues.
