many many hiccups (thousands)

jens55

Latest print,6.5 hours

tool board diagnostic:
hiccups 51778
VIN voltage: min 0.8, current 24.5, max 45.3

Diagnostic for the 6HC for the same print:
Supply voltage: min 23.7, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
hiccups 0
both boards use the same power supply

The print completed despite the nutty numbers.
This is looking (IMHO) more and more like a firmware or hardware design issue

gloomyandy

@jens55 Are your other toolboards showing odd voltage readings? When you run the same test print using a different toolboard I think you said you get the same number of hiccups, but do you also get the same odd voltage readings?

jens55

@gloomyandy
I will run the same print on another tool and will report back

gloomyandy

@jens55 It might be worth also checking the voltage readings on all of the boards that are not printing using m122 b## after the print to see if any of those show odd readings in the same way you have been checking the mainboard.

gloomyandy

@gloomyandy In particular it would be interesting to see what voltages are reported by the board that has been reporting the odd voltages when using one of the other boards for the actual printing.

T3P3Tony

@jens55 said in many many hiccups (thousands):

Another tidbit of information I have discovered - the number of hiccups is directly proportional to the extruder step rate (or number of steps taken during a print).
I decided to try and print with a much lower step rate and a print that had previously generated thousands of hiccups ended up with 105 hiccups. The steps/mm for this extruder should have been 830 but I ran the print at 32 steps/mm.
I don't know what it means but this is the first time I have been able to correlate the hiccups to something.

This makes sense because the hiccups are generated when the processor can't keep up with the requested step rates. In this case its the processor on the toolboard, not the mainboard processor.

Looking at your config.g you have
M92 E25.3:51.875:51.875:51.875

With microstepping you have:

m92

Steps/mm: X: 100.000, Y: 100.000, Z: 3200.000, U: 30.578, E: 404.800:830.000:830.000:830.000

So that tracks.
830 steps/mm is relatively normal for extruders,

Retracts are normally the fasted moves, and looking at the gcode file you shared you are retracting/unretracting at F2700 = 45mm/s (e.g.):

(line10705)

G1 E-1 F2700
G1 X160.818 Y127.668 F18000
G1 X164.656 Y127.668
G1 E1.05 F2700

Normal printing moves, while being a lot of short moves are generally slower extruder movement so going to discount those for now.

45mm/s at 830step/mm = 37350 steps/s or a stepping frequency of ~37Khz which should be achievable by the toolboard processor. We have achieved 45Khz on the 1XD in the past, and that uses the same processor. That said I have not tested with the latest firmware with the changes to input shaping etc so that's a test I need to do when I get a chance.

In the meantime can you do the following tests to help confirm/deny the source of the step rate issue:

Try a vase mode print with no retractions. If you have retractions during the first level then send M122 once the vase mode starts to clear the hiccup counter. then send it again to get a reading
Try the test print you have been using with all the retractions with the extruder microstepping set to 8 or 4, rather than 16.

jens55

@T3P3Tony
Thanks, I will give your suggestions a try once the current test print has finished in roughly 6 hrs.

jens55

@gloomyandy
OK, same print as before but on B23
M122 for the 6HC - no hiccups, good voltages
b21 - no hiccups, good voltages
b22 - no hiccups, good voltages
b23, the tool board that just finished printing:
hiccups 44029,
VIN voltage: min 0.4, current 24.2, max 39.4
b24 - no hiccups, good voltages

jens55

@T3P3Tony

The test print used for the following was in vase mode. Print time was about 14 minutes for each run (hence the much smaller number of hiccups)

Object printed in vase mode but nothing changed on retractions
Hiccups 210
VIN voltage: min 22.8, current 24.2, max 24.2

Same object printed in vase mode with no retractions and no reset after first layer
hiccups 202
VIN voltage: min 22.8, current 24.2, max 24.2

sent M350 E4:4:4:4 I1 to change microstepping on all extruders to 4

same object as previous print, vase mode, no retractions
hiccups 49
VIN voltage: min 22.9, current 24.2, max 24.2

The last object did not print as I didn't correct the steps/mm parameter to compensate for the lower microstepping

Since the number of steps have been reduced substantially (divided by 4), the hiccups went down accordingly.

T3P3Tony

@jens55 thanks, that confirms it's related to the steps, but looks to be extrusion not just retract/unretract. I will have to see if I have similar results on the bench. Can you upload the vase mode file to here so I can use the same file.

jens55

@T3P3Tony
corner clamp.gcode

jens55

Any progress on this issue ?

jens55

I don't know if the hiccup issue is still being looked into or if it has been shelved. I just ran a 6 hr print using 3.5.1 and I am still seeing the same issues. Here is my m122 for the tool board that did the printing. Note the 'hiccups 27753'

m122 b22
Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.1 (2024-04-19 14:42:41)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 3104, free system stack 71 words
Tasks: Move(3,nWait 7,2.2%,85) HEAT(2,nWait 6,8.9%,91) CanAsync(5,nWait 4,0.0%,55) CanRecv(3,nWait 1,2.5%,71) CanClock(5,nWait 1,0.6%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,nWait 6,90.6%,53) MAIN(1,running,38.8%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,122.4%,112), total 266.0%
Owned mutexes:
Last reset 97:43:37 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 3392, slot 1
Software reset code 0x0120 ICSR 0x00000000 SP 0x2000415c Task Freestk 129 bad marker
Stack: 00000544 00022ffc 00019b65 20003134 00016cff 20003134 000163d1 20000ed0 00000000 00000001 00008275 200071c8 200071c8 200071e0 00000000 20000f50 00011647 000223b8 00022474 00021ac8 00019b05 200071c8 200071c8 20000f50 000083ed 200071d8 000009c7
Driver 0: pos 0, 830.0 steps/mm, standstill, SG min 0, read errors 10, write errors 0, ifcnt 225, reads 7857, writes 3, timeouts 32, DMA errors 0, CC errors 0, failedOp 0x6a, steps req 0 done 73029969
Moves scheduled 394767, completed 394767, in progress 0, hiccups 27753, segs 33, step errors 0, maxLate 1 maxPrep 495, maxOverdue 136, maxInc 71, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 1.00
Peak sync jitter -1/4, peak Rx sync delay 272, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 17.7, current 24.4, max 24.5
MCU temperature: min 32.8C, current 36.1C, max 54.4C
Last sensors broadcast 0x00000004 found 1 26 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 1831836, send timeouts 0, received 2249695, lost 0, errs 0, boc 0, free buffers 18, min 17, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 561, adv 35643/74674
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 0, contentions 0, other errors 0
=== Filament sensors ===
Interrupt 5726621 to 0us, poll 5 to 2912us
Driver 0: ok

Phaedrux

Can you share a video of the printer in action?

jens55

@Phaedrux, while not impossible, I don't really know what I am doing. Could you give me a hint as to what you expect to see? There is no apparent (at least to me) difference in printer behaviour between a printer that shows hiccups vs one that doesn't produce hiccups. I don't know if maybe one would see choppy motion for example ... but there is no hint of choppiness.