Duet2 CPU struggling to keep up with this gcode?

Edgars Batna

I've been hunting down a problem as described in my previous threads:
https://forum.duet3d.com/topic/10435/still-can-t-print-due-to-extruder-steppers-skipping-steps
You may scroll to the bottom and watch the video or read all of the other thread that is linked in the above thread.

It turns out that me, the clumsy thickhead, was searching in the wrong direction the whole time. It appears that instead of pressure advance having a problem it's the kinematics computations that the Duet is struggling with, which in turn break the pressure advance.

The more detailed the gcode, the more the movement starts stuttering and grinds to a near complete halt if I increase the resolution. Instead of requested 120mm/s it's going at 8mm/s. I've tested all this without extrusion and mesh bed compensation using version 2.03 and 2.02. If enabled, PA computes impossibly high compensation factors because the printer halts inbetween.

Video: https://youtu.be/1NALZdyvBPc

Here's a gcode where any move above ~40mm/s starts stuttering: 0_1564074914986_PA tiny segment test EccentricGearTest.gcode
Here's a much finer gcode of near exact same moves where the Duet can't seem to populate the DDA ring in time and runs stuttery at 8mm/s (depicted in the video above): 0_1564074998392_EccentricGearMacroProcessedSuperfineNoE.gcode

I've run M122 multiple times but get only a tiny fraction of underruns in comparison to move count: 0_1564075249093_duetlog.txt

I've put some logging into the code and I can see that the DDA ring is empty most of the time. If there are moves then it still stutters along very slowly.

I suspect the CPU is struggling with this print job, but I don't see where the heavy computations are happening. If I adjust various things like DDA ring length, minimum prepared time, minimum IRQ interval then nothing changes.

If this is indeed CPU struggling, then the symptoms are a bug. I expect the Duet to slow down instead of stuttering and causing PA to lose steps.

It's a CoreXY with an ultra-light print gantry+head and here's my config.g: 0_1564076201550_config.g

Config and printer appear just fine. It has printed quite a bit now. I've been spraying logging around DDA::Init and DDA::Prepare for the last few months.

What could be taking so much time or why such a slow CPU was chosen, if it's the case?

Phaedrux

M98 P"0:/macros/EccentricGearMacroProcessedSuperfineNoE.gcode"

Is this how you're starting prints? I'm not sure that behaves the same as https://duet3d.dozuki.com/Wiki/Gcode#Section_M32_Select_file_and_start_SD_print
I really don't know if it matters, but it seems an odd way to start a print.

@edgars-batna said in Duet2 CPU struggling to keep up with this gcode?:

I've tested all this without extrusion and mesh bed compensation

Except that your M122 shows mesh being active? I notice you have G29 S1 in config.g, which is not recommended because the heightmap is being loaded before Z0 has been determined. But it also means it's being reactivated every time to power up.

The video looks exactly like the behaviour you'd see from a print with too low Z/E jerk (M566 Z150 E900) to accommodate the movement speed and the mesh Z compensation requirements. Perhaps exacerbated by high pressure advance value (M572 D0:1 S0.6). I also notice you have a very tight mesh (M557 P5) and are tapering the compensation (M376 H20) which can have an effect on extrusion as well. See https://duet3d.dozuki.com/Wiki/Gcode#Section_M376_Set_bed_compensation_taper

Just a few things I noticed.

Edgars Batna

@phaedrux said in Duet2 CPU struggling to keep up with this gcode?:

M98 P"0:/macros/EccentricGearMacroProcessedSuperfineNoE.gcode"
Is this how you're starting prints? I'm not sure that behaves the same as https://duet3d.dozuki.com/Wiki/Gcode#Section_M32_Select_file_and_start_SD_print
I really don't know if it matters, but it seems an odd way to start a print.

This happens when I run a macro using web control.

@edgars-batna said in Duet2 CPU struggling to keep up with this gcode?:

I've tested all this without extrusion and mesh bed compensation

Except that your M122 shows mesh being active? I notice you have G29 S1 in config.g, which is not recommended because the heightmap is being loaded before Z0 has been determined. But it also means it's being reactivated every time to power up.

The video looks exactly like the behaviour you'd see from a print with too low Z/E jerk (M566 Z150 E900) to accommodate the movement speed and the mesh Z compensation requirements. Perhaps exacerbated by high pressure advance value (M572 D0:1 S0.6). I also notice you have a very tight mesh (M557 P5) and are tapering the compensation (M376 H20) which can have an effect on extrusion as well. See https://duet3d.dozuki.com/Wiki/Gcode#Section_M376_Set_bed_compensation_taper

Just a few things I noticed.

This is my "default config.g", but I've already tested that disabling all of them has no effect. PA, bed compensations and extrusion have no effect for me in the "superfine" test macro.

Would you please try the test gcode (EccentricGearMacroProcessedSuperfineNoE.gcode)?

deckingman

@edgars-batna I just ran that g code which ran like a pile of crap. But then I looked at the file and realised why. How on earth was that file generated and what do you expect?

A quick gander shows that it's made up of segments mostly in the order of 0.004mm. I note that you have your steps per mm set to 160 (presumable you are using 0.9 degree motors). So a segment of 0.004 mm would be 0.64 of a micro step. And as you can't have a fraction of a micro step, that is going to get rounded up to 1 micro step (maybe) unless RRF always rounds down in which case you'll get zero micro steps. But a segment of 0.003 (and there are plenty in that file) would be 0.48 of a micro step so rounded down to zero. The firmware is likely ignoring (or rounding down to zero) half of those tiny moves which are fractions of a single micro step. Or it's rounding up those fractions of a micro-step that are just about. So you either get no movement or twice the asked for movement. Hence all the stuttering.

One thing is for sure. Given that forward filament motion is likely to be single digit percentages of carriage motion, then there is no way that you'd ever print that.

You could try setting micro-stepping to 256x which will help to negate the rounding errors but I'm not going to waste any more time on it.

Edgars Batna

@deckingman Thanks for looking at it, but setting to x256 doesn't affect it as well.

The "superfine" gcode was generated from the first gcode using spline interpolation that I did for testing. The whole point is that it's an exaggerated test to demonstrate the problem. I'm getting this problem in the first gcode above ~40mm/s... I don't expect the Duet to run the "superfine" gcode without struggling. I expect it to at least move consistently.

dc42

Please confirm that you are printing this file from the micro SD card on the Duet, not from an external SD card or over USB.
After running that GCode, please run M122 and post the report here. The hiccup count and underrun counts may tell us what is happening.

Edgars Batna

@dc42

I'm running off the micro SD card on the Duet.
0_1564160839730_M122SuperfineNoE.txt

This time M122 reports no underruns but the movement was stuttery as in the video O.o

I just tried 1/10th resolution and it still wasn't smooth, but not as bad as this superfine gcode.

dc42

There is a limitation on how many GCode commands any controller can process per second. It should be higher on the Duets (especially the WiFi/Ethernet) than on 8-bit controllers. So "superfine" GCode files are always going to be problematic. If you are using a 0.4mm nozzle, then there is no point in using segment lengths smaller than about 0.1mm.

If the print head is still following the commanded path, then IMO it's not a bug. If it didn't follow the commanded path, that would certainly be a bug.

Edgars Batna

@dc42 But in my test this stuttering causes PA to act weird and miss steps due to start/stop. Can't the Duet detect this condition and slow down? If I turn the speed dial all the way back, it gets better for example. The problem here is that this condition can happen anywhere at any speed. Just a few tiny moves somewhere, maybe a tiny rounded detail, but it may do enough to clog the extruder since they start losing steps and going backwards. Needless to say, they also grind the filament and may snap it because PA does adjustments on all of these tiny moves rapidly.

Here's a 1/10th resolution test for reference and it sounds as if some tiny moves are done with insane acceleration: 0_1564168958029_EccentricGearMacroProcessedFineNoE.gcode
Video without extruders: https://youtu.be/hM5WteE87UA
Video with extruders: https://youtu.be/2Ly4vLBV3YI

Theoretically it could be possible to offload some of the GCode parsing and motion planning to another device like my PC, but that's a project larger than the printer itself. Is any of this feasible?

deckingman

@edgars-batna I still don't get it. Your 1/10 resolution gcode file is still mostly made up of segment lengths that are 0.05mm and even less. Why? What's the point? If you want that fine a detail, you should be using something like a sub 0.1mm nozzle and filament that is maybe 0.5m diameter (and either bloody good eyes or a magnifying glass to see it).

Edgars Batna

@deckingman It's not about resolution but narrowing down on the problem. As soon as the Duet runs out of breath strange things happen. Those crazy files just exaggerate the problem to demonstrate it. The problem occurs even on a completely normal file like this: 0_1564172383619_PA tiny segment test EccentricGearTest.gcode

The average distance in the "fine" file is 0.1. At least that's what I've programmed the interpolation to do. I can change this to any value actually.

bot

Well, there is indeed a reason to use detailed gcode: to get detailed prints. A tiny flick of the nozzle goes a long way in dispensing plastic where it needs to be or avoiding areas it doesn't.

That said, I also feed insanely detailed gcode into the duet and experience no such issues.

I didn't study your settings or look at your gcode, but you must be asking the printer to do something that is physically unreasonable for it to do. Pressure advance asks your extruder to do some pretty funky things, if you let it.

Edgars Batna

@bot I'm definitely not asking anything impossible from my printer. If anything, I'm disappointed to run into such issues in 2019 or even using "8bit" and the year in teh same sentence... heh... Duet is, after all, not cheap.

bot

@edgars-batna You are likely "asking" your extruder to move faster than is physically possible for it to do while gripping filament. Intentional or not.

I'm pretty sure I also have segment lengths in the nanometer range, but I don't see a problem. Again, I haven't studied your settings so I'm just saying take a look at everything, and dial some things WAY back, slow it down, and see when you can get it to work nicely... that'll be your max-capability, no?

Phaedrux

Can you describe your extruders? They look like 2 nema23s with planetary gearboxes running in tandem.

deckingman

@bot @Phaedrux Have you tried running his gcode file like I have?
I get the same bad behaviour but have never experienced anything like that with anything I've ever printed, including "real world" printing at 300 mm/sec using 5 extruders.
That's why I believe the problem lies with the file.
Strongly advise that you try printing his file, before looking elsewhere for the cause.

Edgars Batna

@bot said in Duet2 CPU struggling to keep up with this gcode?:

@edgars-batna You are likely "asking" your extruder to move faster than is physically possible for it to do while gripping filament. Intentional or not.

Have you looked at the videos? There is no filament inserted. I've spent so much time on this, all the while changing extruders, hotend, PTFE tubes, motors, configuration that I'm 99.9999999% sure it's not my printer.

@phaedrux said in Duet2 CPU struggling to keep up with this gcode?:

Can you describe your extruders? They look like 2 nema23s with planetary gearboxes running in tandem.

It doesn't matter how I connect them or what microstepping I use, it's consistently reproducible with the files provided. Frankly, if I increase microstepping then the whole thing runs way too slow (which is a bummer, that's why I think we should have been done using slow CPUs for CNC machines). Currently I've got 2x 2A Nema17 with 5:1 gears, but, as I said, I've tried it all already. No combination fixes the problem. It's partly documented in other threads.

Phaedrux

@edgars-batna said in Duet2 CPU struggling to keep up with this gcode?:

it's consistently reproducible with the files provided.

Garbage in garbage out?

Do you have any actual gcode files produced by a common slicer that exhibits this to test?

I just printed your gcode file described as "completely normal" 0_1564172383619_PA tiny segment test EccentricGearTest.gcode

It sets acceleration to 10000 and then starts hammering.

Here ya go: https://www.dropbox.com/s/y6jict6tpgdfvxr/IMG_6027.mp4?dl=0

I added my start and end gcode so it would actually print. The acceleration and print speeds are pretty insane for such a part, but it didn't stutter at all.0_1564268355944_config.g

I think your print settings are unusual to say the least?

Edgars Batna

@phaedrux Thanks for trying it out. By no means it's all garbage. Have you looked at the other thread, linked in the beginning? It fails at completely normal prints and so far all indicates that this is the culprit with PA enabled, which gets even worse with higher PA values.

The "PA test" macros are from 2nd layer onwards. Could you remove filament, set PA to 0.6 and run the "PA tiny segment test EccentricGearTest" file? The extruders should start acting up. I just ran the same with 0.035 PA (as per your config) and they still acted weird at certain points but not as much because there are no sharp turns in this gcode.

Here's the actual printable gcode: 0_1564303068509_EccentricGearTest 0 0 volumetric.gcode. It does print, but there the PA is only slightly shy of skipping steps. With slightly higher move count the PA commands the extruders to skip because the Duet can't keep up. The extreme demonstration is contained within the "fine" and "superfine" macros.

Here's the "fine" macro with E: 0_1564304231040_EccentricGearMacroProcessedFine.gcode
Here's the "superfine" macro with E:
0_1564304239370_EccentricGearMacroProcessedSuperfine.gcode
WARNING: remove filament before trying
IMPORTANT: enable PA and try different low and high values when trying

The macros are for quick reproduction and are not printable. By "normal" I referred to the remarks by other users regarding the segment count and resolution.

EDIT #342354: Had to resort to contacting Akismet so that they stop blocking my edits...