Another crack at extruder problems

Edgars Batna · 16 Mar 2020, 18:16

@bot @Phaedrux This is an acknowledgment of attempted unintentional thread hijack, sorry.

This is a continuation of the big old threads about extruder skipping steps. For those up for a long story (in reversed chronological order):

@dc42 My case is that the Duet 2 may run out of CPU with PA set on high value on small segments. My analysis with poor man's debugging from last year showed that DDA ring gets exhausted and new GCodes cannot be processed fast enough to replenish it, hence the extruders receiving stuttery steps and eventually losing them. The GCode parsing takes longer than the step execution. On my printer with current settings it occurs nearly always on Gyroid infill mostly somewhere above 60mm/s. Reducing speed just reduces the CPU load and probability of this issue occuring, it does not fix the problem completely. I tried alleviating the problem but found out I'd need multiple GCode states and buffers, which would fill up the RAM and even then the CPU would be too slow. Additionally, there is some error accumulating and causing extruders to turn backwards. Read the information below first, tho.

@bot said in Enhancing pressure advance:

For @Edgars-Batna, I would suggest trying a much lower E jerk value.

With M566 E100:100 the printer is much more silent, almost purrs like a cat, but this is mighty slow on them corners. But again, it's only slow if pressure changes... I had to also change PA to 1.5 for the start & end segments to be tidy.

@deckingman said in Enhancing pressure advance:

@Edgars-Batna Just a random(ish) thought but that combination of lowish extruder acceleration and especially that low "jerk" might be limiting the pressure advance which is why you need such a high value. What happens if you double the extruder jerk and acceleration but halve the PA? But then IRC, don't have your extruders in push-pull configuration? In which case, I'm wondering if they might get a bit out of sync when PA is applied so end up fighting each other? Can you run with just one extruder to test? Just another random thought.....

If I increase the E jerk then the printer sounds a like a self-hammering ticking nuclear bomb. The extruders are 5cm apart, it's basically a multi hobb configuration. I've tried single motor before and saw no difference regarding the issue.

Now, all things considered, I'm running my own 2.02.3 firmware with some changes. If I install official 2.02+ then the printer explodes spectacularly, mostly on first layer. Right now I don't know what next. I could tidy up my local changes and put up a patch, but how comes I'm the only one with the problem...

arhi · 16 Mar 2020, 18:55

@Edgars-Batna said in Another crack at extruder problems:

but how comes I'm the only one with the problem...

Random ideas

% of ppl using high PA is low
% of ppl using moves under .1mm is also low

combining [1] and [2] gives only you

Edgars Batna · 16 Mar 2020, 19:02

@arhi said in Another crack at extruder problems:

@Edgars-Batna said in Another crack at extruder problems:

but how comes I'm the only one with the problem...

Random ideas

% of ppl using high PA is low

% of ppl using moves under .1mm is also low

combining [1] and [2] gives only you

I know, but see it from another perspective. We're talking a not so big program, which reinvents the wheel at a few up to nearly all places just to accommodate a microcontroller. This should not be necessary anymore... I do not doubt in capability of Duet's programmers, just that the method is getting older by the day and it's infuriating on my end. Just what are we trying to save here? Cents on the CPU? The printer already runs at multiple hundreds of Watts and any reoccurring error wastes kilos of filament. Please, not the rant again... I'd rather focus how this can be fixed. Personally I'd never use a CPU below 4GHz and 8GB RAM these days.

Phaedrux · 16 Mar 2020, 19:28

Can you post your config and a sample gcode file that produces the issue?

Can you provide some details on the extruder arrangement. Motors, etc.

I know lots of that info is in the other threads but having the latest info all together here would be helpful.

At this point, development on RRF2 is slowed down to emergency fixes only. RRF3 is going to be the way forward. Have you tried testing RRF3 yet? Or even 2.05?

bot · 16 Mar 2020, 20:19

I have been printing at high speeds, with tiny segments and have not experienced this issue.

You have to face reality: PA applies an artificial, instant adjustment to the extruder position. You are selecting a combination of settings that is asking the extruder to move fast enough to sound crazy. Whatever changes you made to the firmware must have reduced the magnitude or frequency of your settings requesting moves that sound like hammer drills.

I'm not sure why you think you are encountering limits with parsing g-code or the like... I don't think that's the case. I think you're simply (like I've said in other threads a year ago) asking your extruder to do things it physically can't. Period.

arhi · 16 Mar 2020, 20:57

@Edgars-Batna said in Another crack at extruder problems:

I know, but see it from another perspective. We're talking a not so big program, which reinvents the wheel at a few up to nearly all places just to accommodate a microcontroller. This should not be necessary anymore...

I dislike SAM MCU's but I love ARM core (both cortex m3, m4, m7 .. ), I also did not look in-depth into the duet source... The general view I made some time ago I did not like (tbh I don't remember why), and looking at few available cortex firmware's around I really only really like the smoothieware design, but it comes with a lot of limitation and also lacks bunch of features (no PA in any way there for e.g.)

Personally I'd never use a CPU below 4GHz and 8GB RAM these days.

Then the concept like Klipper is what you should look at. You ran all the calculations on the Linux computer that don't even need to have a real-time kernel and only the basic stepping is done by the MCU providing precise timings. You can run the basic setup on some SBC or you can use your 24 core 512G RAM desktop if you like

sebkritikel · 17 Mar 2020, 01:27

Does this quick video (your EccentricGearTest 0 0.gcode file) represent the issues you face while printing (including the movement of the print head)?
https://www.youtube.com/watch?v=J-_RAQJW1kk&feature=youtu.be

I incorporated these lines into the GCODE file in an attempt to emulate your setup.
M207 S3 R0.0125 F20000 Z0.5 ;firmware retraction
M572 D0 S0.5 ; pressure advance (S) in seconds

Edgars Batna · 17 Mar 2020, 20:11

@sebkritikel said in Another crack at extruder problems:

Does this quick video (your EccentricGearTest 0 0.gcode file) represent the issues you face while printing (including the movement of the print head)?
https://www.youtube.com/watch?v=J-_RAQJW1kk&feature=youtu.be

I incorporated these lines into the GCODE file in an attempt to emulate your setup.
M207 S3 R0.0125 F20000 Z0.5 ;firmware retraction
M572 D0 S0.5 ; pressure advance (S) in seconds

Thanks for trying it out. Yes, it sounds much like what I got, but less prevalent (probably due to different mechanics). For reference, the GCode:
EccentricGearTest 0 0 volumetric(1).gcode

The parts where we hear the "crackling" is basically undefined behavior.

In any case, here's a new vid of mine: (ignore the build crustiness): https://youtu.be/CddfoiOETS4

The loudness depends on the mechanics. I got these 3:1 reduction gears with a little backlash, so the mechanics sound like, I dunno, welding metal or percussion drill depending on listener position.

This is an extreme demo of a problem - Duet should slow down, but there's nothing implemented for it and it's actually non-trivial for the firmware. The worst that can happen is that the extrusion runs backwards enough to destroy a print.

In case someone is wondering about what the hell is going on in my build with the wood blocks, reduction gears, cables, paper, roll bearings etc, I went to lengths to ensure the machine works well, including no skips, no clogs, no grinding, true bed leveling and general well being in the mechanics before posting these topics.

@arhi said in Another crack at extruder problems:

I dislike SAM MCU's but I love ARM core (both cortex m3, m4, m7 .. ), I also did not look in-depth into the duet source... The general view I made some time ago I did not like (tbh I don't remember why), and looking at few available cortex firmware's around I really only really like the smoothieware design, but it comes with a lot of limitation and also lacks bunch of features (no PA in any way there for e.g.)

Then the concept like Klipper is what you should look at. You ran all the calculations on the Linux computer that don't even need to have a real-time kernel and only the basic stepping is done by the MCU providing precise timings. You can run the basic setup on some SBC or you can use your 24 core 512G RAM desktop if you like

Thanks, I'll check it out. I got a nearly 10 years old 4 Core 3GHz, 16GB RAM lying around without purpose and more scattered 16GB RAM sticks around the house, I could throw it all at the printer with enough force for it to weld to the Duet some time in the future.

@Phaedrux said in Another crack at extruder problems:

Can you post your config and a sample gcode file that produces the issue?
config_17032020.g
Can you provide some details on the extruder arrangement. Motors, etc.
2 extruders arranged 90 degrees to each other, driven by separate drivers in 50:50% mixing mode. Also, see the video.

At this point, development on RRF2 is slowed down to emergency fixes only. RRF3 is going to be the way forward. Have you tried testing RRF3 yet? Or even 2.05?

I think I tried up to 2.04, then gave up, because the firmware didn't actually get more efficient from my observations and I couldn't be bothered to unclog the printer every time.

sebkritikel · 17 Mar 2020, 21:55

With that GCODE test we definitely have a lot going on.

Regarding the filament extrusion running backwards - It is my understanding (from what I have read here) that based on your (generic your, not saying specifically your system) system, it is perfectly acceptable to have the extruders running backwards during certain moves based on PA settings. However, 100mm is not normal, and I do NOT believe that is related to (or more specifically, DIRECTLY caused by) pressure advance. In my experience with BCN3D Sigmax machines, with bowden tubes of 865mm, a jam (or flow restriction) in the nozzle results in a high pitched squeal from the extruders, with the filament jumping backwards by a significant amount. The dual drive extruders on these machines are capable of compressing the fillament and stretching the tubes during a clog/jam, and when this back pressure is able to overcome the extruder motor. Based on your print temperatures (215°c or so right) I could definitely see jamming or clogging happening, but I'm sure you've looked at this.

Still a lot more to unpack but to be honest, I feel like its working as intended, that high value of PA should sound like gunshots going off (especially with NEMA23s). I'm far from an expert though!

Edgars Batna · 19 Mar 2020, 17:04

@sebkritikel said in Another crack at extruder problems:

With that GCODE test we definitely have a lot going on.

Regarding the filament extrusion running backwards - It is my understanding (from what I have read here) that based on your (generic your, not saying specifically your system) system, it is perfectly acceptable to have the extruders running backwards during certain moves based on PA settings. However, 100mm is not normal, and I do NOT believe that is related to (or more specifically, DIRECTLY caused by) pressure advance. In my experience with BCN3D Sigmax machines, with bowden tubes of 865mm, a jam (or flow restriction) in the nozzle results in a high pitched squeal from the extruders, with the filament jumping backwards by a significant amount. The dual drive extruders on these machines are capable of compressing the fillament and stretching the tubes during a clog/jam, and when this back pressure is able to overcome the extruder motor. Based on your print temperatures (215°c or so right) I could definitely see jamming or clogging happening, but I'm sure you've looked at this.

I've taken care of all that multiple times.

Oh well, I'll bite the bullet again and update to latest firmware, fingers crossed. 2.05.1 first and if that goes well, then RRF 3.

bot · 19 Mar 2020, 17:24

@Edgars-Batna 2.05.1 seems fine, but I would not go to 3.0 if you are "not happy" with how your printer acts with 2.05.1. You'll have so many more "unknowns" that it will be even worse to try and troubleshoot.

Edgars Batna · 19 Mar 2020, 17:26

@bot said in Another crack at extruder problems:

@Edgars-Batna 2.05.1 seems fine, but I would not go to 3.0 if you are "not happy" with how your printer acts with 2.05.1. You'll have so many more "unknowns" that it will be even worse to try and troubleshoot.

Quick test with the EccentrigGearTest GCode on 2.05.1 actually appeared to run way smoother. Will run more tests with it. The "problems" are so deceptive I'll need a few kilos of filament before knowing if it actually works.

bot · 19 Mar 2020, 17:32

@Edgars-Batna I know how it is man. The past two months I've been printing the same stuff non-stop trying to converge on the perfect balance of speed vs quality... and it's driving me insane. I'm getting close to accepting a set of values.

It's all about compromise. Choosing values to optimize for and then accepting the reality that nothing will be perfect. I mean... I can print an absolutely PERFECT print... it'll just take days longer than ~~it should~~ I would like.

Edgars Batna · 20 Mar 2020, 14:00

@bot said in Another crack at extruder problems:

@Edgars-Batna I know how it is man. The past two months I've been printing the same stuff non-stop trying to converge on the perfect balance of speed vs quality... and it's driving me insane. I'm getting close to accepting a set of values.

It's all about compromise. Choosing values to optimize for and then accepting the reality that nothing will be perfect. I mean... I can print an absolutely PERFECT print... it'll just take days longer than ~~it should~~ I would like.

I just printed a small test figurine and it was PERFECT at 60mm/s and 1.25 PA. I went on to print a bigger piece that's been failing and it clogged itself on gyroid infill, as it appears. It's a print with very long 120mm/s moves and it appears as if extrusion was inconsistent around beginning of perimeters, but went to be really consistent in the middle of the perimeter moves.

Afterwards I ran M122 and saw peculiar things in the log: console.txt

Running 2.05.1.

This and other variations where config.g does not make it to its destination keeps happening:
19.3.2020, 21:17:35: : Error: Failed to rename file or directory 0:/sys/config.g to /sys/config.g.bak

Can we please get screw terminals already, I'm tired of these crimp connections:
20.3.2020, 00:32:41: : Warning: motor phase A may be disconnected reported by driver(s) 1
Warning: motor phase B may be disconnected reported by driver(s) 1

What happened here, I saw my printer reset yesterday evening for no apparent reason, and here it is, didn't know a heat task can get stuck:
Last software reset at 2020-03-19 19:29, reason: Heat task stuck, spinning module Platform, available RAM 10536 bytes (slot 1)
Software reset code 0x40a0 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80f BFAR 0xe000ed38 SP 0x200049ec Task 0x4e49414d
Stack: 0044d797 0043f178 01000000 3f800000 40418666 00000000 00000000 3ff78a38 41400000 3e178897 3e1cd04f bd564f13 3e3a336f 3e638e76 3e924953 3b33684c 395debbd 3f800000 3f800000 20000011 0043f01b 00000000 00424ba3

Is this alright:
Slowest loop: 143.02ms; fastest: 0.07ms

How about this, does it mean something? This is the first firmware that I see reporting hiccups:
Hiccups: 5120, FreeDm: 160, MinFreeDm: 113, MaxWait: 285267ms

Post mortem of the print: a whole layer is missing. Multiple blobs in layers below. Then the printer recovered and printed 3-4 layers mid air before I came over to check on it. Is it trolling me? Should I drive it to a recycling dump?

bot · 20 Mar 2020, 15:32

@Edgars-Batna The hiccups are because you are running too fast.

Your PA/Accel/Jerk/Feedrate values are commanding too much movement. It's too much to physically handle (extruder) and it's too much for the pulse generation.

That's not to say that it's unable to read the gcode lines fast enough, but that it can't generate steps fast enough for what you are asking.

Edgars Batna · 20 Mar 2020, 15:59

@bot said in Another crack at extruder problems:

@Edgars-Batna The hiccups are because you are running too fast.

Your PA/Accel/Jerk/Feedrate values are commanding too much movement. It's too much to physically handle (extruder) and it's too much for the pulse generation.

That's not to say that it's unable to read the gcode lines fast enough, but that it can't generate steps fast enough for what you are asking.

I understand, BUT how should one know when this will happen? The machine should have some sort of logic in place to slow down or state clearly that a section is too fast when this happens. There is absolutely no indication that it goes too fast until it's too late and kilos of filament are wasted. I'm not sitting 40 hours in front of a print, and, with all the fine tuning required, it's harder to know where it's too fast than it should be.

bot · 20 Mar 2020, 16:58

@Edgars-Batna The hiccups are exactly the firmware indicating and dealing with the problem.

The hiccups are delays of 50 microseconds in order to attempt to delay enough to have time to catch up.

Essentially, the firmware is acting as best as it can given all the circumstances. Now that you know there is a problem it is up to you to slow things down when there is a problem.

I suggest you periodically send M122 during a long print to see when the hiccups are occuring, if you don't wish to try slowing down random parts first.

But let's talk about pressure because I think this proposed change could actually help a lot.

I think the proposed change to pressure advance could basically only be placed where a retraction is -- in between the retract and unretract movements.

This is essentially dynamic retraction but for only a specific circumstance: feedrate change of print moves.

Edgars Batna · 20 Mar 2020, 17:31

@bot said in Another crack at extruder problems:

@Edgars-Batna The hiccups are exactly the firmware indicating and dealing with the problem.

The hiccups are delays of 50 microseconds in order to attempt to delay enough to have time to catch up.

Essentially, the firmware is acting as best as it can given all the circumstances. Now that you know there is a problem it is up to you to slow things down when there is a problem.

I suggest you periodically send M122 during a long print to see when the hiccups are occuring, if you don't wish to try slowing down random parts first.

But let's talk about pressure because I think this proposed change could actually help a lot.

I think the proposed change to pressure advance could basically only be placed where a retraction is -- in between the retract and unretract movements.

This is essentially dynamic retraction but for only a specific circumstance: feedrate change of print moves.

I see what you mean, but I suggest at least a list of offsets in the gcode file so that one can analyze this better.

In any case, your proposal is reasonable, now we need a firmware with this change implemented to test it.

dc42 · 16 Sept 2020, 07:49

@Edgars-Batna, I apologise for the long delay in returning to this thread. COVID-19 resulted in a huge increase in forum traffic, and at the same time one of our support engineers had to greatly reduce the hours he works for us in order to look after children who were no longer able to go to nursery. The forum is now back to a more normal level of traffic.

I'm glad to hear that upgrading the firmware to 2.05.1 solved some of the problems.

Regarding hiccups, please post your config,g file so that we can see whether you are using appropriate microstep settings.

Regarding screw terminals, I'm sorry, we will not revert to them because OEMs absolutely hate them. Connecting wiring looms to screw terminals is an expensive (in time) and error-prone process, and screw terminals can loosen. Crimp connections done properly with an appropriate crimping tool and an appropriate wire gauge are reliable.

Regarding gyroid infill, I note that in a previous thread you seem to think that RRF can't parse the GCodes fast enough in some situations with pressure advance applies, because the print slows down. Does it only slow down when using pressure advance? Using PA causes only a small increase in processor load. OTOH gyroid infill results in lots of very short segments of GCode. We know that in these situations, some slicers fails to maintain a consistent extrusion rate, and that plays havoc with PA. If that's the problem, the fix is for the slicer to maintain a consistent extrusion rate. This may be as simple as adding one more decimal place to the extrusion distance in the GCode. One of our users has already done this in a fork of PrusaSlicer.

HTH David

Edgars Batna · 16 Sept 2020, 08:14

Based on what I've found out so far, the next issue to tackle is the over-extrusion with PA enabled. There's another thread on this which should be tackled before tackling this one: https://forum.duet3d.com/topic/18412/rrf-2-03-pressure-advance-causes-20-overextrusion/109

For consistent extrusion to work, there has to be a decent resolution in the GCode. This increases the over-extrusion from 3% to multiple times more (no fancy tools at hand, but optically it's over 50%, it used to be even worse in older firmwares). That's exactly the problem, is that it's not consistently just 3%. In the other thread it started off with 20% and with various changes it dropped to 3%, but it's just a single print, not a general reading.

@dc42 said in Another crack at extruder problems:

I'm glad to hear that upgrading the firmware to 2.05.1 solved some of the problems.

The extruders run way better indeed, but there's still room for improvement.

Regarding hiccups, please post your config,g file so that we can see whether you are using appropriate microstep settings.

I've followed every single recommendation multiple times already. Increasing E resolution to 64 does next to nothing.

config_01092020.g

Regarding screw terminals, I'm sorry, we will not revert to them because OEMs absolutely hate them. Connecting wiring looms to screw terminals is an expensive (in time) and error-prone process, and screw terminals can loosen. Crimp connections done properly with an appropriate crimping tool and an appropriate wire gauge are reliable.

It's alright, I've learned to live with it.

Regarding gyroid infill, I note that in a previous thread you seem to think that RRF can't parse the GCodes fast enough in some situations with pressure advance applies, because the print slows down. Does it only slow down when using pressure advance? Using PA causes only a small increase in processor load. OTOH gyroid infill results in lots of very short segments of GCode. We know that in these situations, some slicers fails to maintain a consistent extrusion rate, and that plays havoc with PA. If that's the problem, the fix is for the slicer to maintain a consistent extrusion rate. This may be as simple as adding one more decimal place to the extrusion distance in the GCode. One of our users has already done this in a fork of PrusaSlicer.

Part of changes in that fork are based on my recommendations. I use that fork exclusively since it got available.

The slowdown is the place to analyze the issue, as I believe it's related to over-extrusion. It's a crash/smoke test - just slap an insanely detailed GCode onto the RRF and see what breaks. All of it needs fixing.

I also have my own RRF 2.0 fork where I tried to fix the over-extrusion for my printer and add some experimental features. It might be hard for you to navigate for fixing just this problem, though: https://github.com/mdealer/RepRapFirmware/tree/simple_dynamic_retraction . The issues are not 100.0000000% fixed there, but alleviated to a usable degree on my printer.