High number of hiccups during basic moves

MrSteve920

Hi,

I'm having some trouble with my delta using a Duet WiFi 1.01. Recently my prints have been experiencing arbitrary layer shift throughout the course of any prints. I checked all of the usual suspects such as loose pulleys and belts but all of those were fine. After seeing a few other unrelated posts, I decided to review the output of M122. With a small print (less than 1 hour) I had over 60,000 hiccups reported. Consistently, I am able to get the printer to report about ~15 hiccups just from diving down 50mm from home and re-homing. Changing microstep values from x64 to x32 did drop the number of hiccups but the printer has always operated using x64 and it has never had issues like it is experiencing now. Running firmware 2.02 release, but I did go back to 2.01 to see if the problem persisted and it did. This issue only started about a week ago. Let me know if I should provide any additional debug info.

Config.g:

; Configuration file for UltiBots D300VS Delta 3D Printer

; Communication and general
M111 S0                             	; Debug off
M550 D300VS				; Machine name and Netbios name (can be anything you like)
M551 XXXXX				; Machine password (used for FTP)
;*** If you have more than one Duet on your network, they must all have different MAC addresses, so change the last digits
M540 P0xBE:0xEF:0xDE:0xAD:0xFE:0xEF 	; MAC Address
;*** Wifi Networking
M552 S1					; Enable WiFi

M555 P2                           	; Set output to look like Marlin

M575 P1 B57600 S1			; Comms parameters for PanelDue

G21                                 	; Work in millimetres
G90                                	; Send absolute coordinates...
M83                                 	; ...but relative extruder moves

; Axis and motor configuration
M569 P0 S0				; Drive 0 goes forwards (X tower)
M569 P1 S0				; Drive 1 goes forwards (Y tower)
M569 P2 S0				; Drive 2 goes forwards (Z tower)
M569 P3 S0				; Drive 3 goes BACKWARDS (Extruder)

M574 X2 Y2 Z2 S1			; set endstop configuration (all endstops at high end, active high)

; Old calibration values from Sept. 2018. Config override is using newer diagonal rod length calculation.
M665 L377.253 R213.682 H442.002 B150.0 X-0.365 Y-0.282 Z0.000	; set delta radius, diagonal rod length, printable radius and homed height. L is diagonal. R is delta radius, H is homed height, XYZ degree offset.
M666 X-1.087 Y0.609 Z0.478 A0.00 B0.00				; put your endstop adjustments here, or let auto calibration find them. 

M579 X1 Y1 Z0.994			; Scale cartesian axis calibration. X was 0.997 Y was 0.999 12/21
M350 X64 Y64 Z64 E16 I1			; Set microstepping to 32 for X, Y and Z and 16 for extruder stepper with interpolation
M92 X800 Y800 Z800			; Set axis steps/mm
M906 X1000 Y1000 Z1000 E450		; Set motor currents (mA)
M201 X1000 Y1000 Z1000 E1000		; Accelerations (mm/s^2) . Original was 1000. New was 3000.
M203 X20000 Y20000 Z8000 E3600		; Maximum speeds (mm/min)
M566 X1200 Y1200 Z1200 E300		; Maximum instant speed changes mm/minute. Go as low as 250 if ringing is still present. Original was 1200. New was 370.
;M572 D0 S0.07				; Pressure advance

; Fans
;M106 P1 T50 S255 H1 			; Set hotend heatsink FAN1 thermostatic control at 50°C. Disabled due to damaged MOSFET. Hotend fan reassigned to always on header.
M106 P0 H-1 S0 C"Part Cooling Fan" X0.7	; Part cooling fan
M106 P2 H-1 S255 C"LED Lighting"	; LEDS on PWM port

; Thermistors
M305 P0 T100000 B3950 R4700 L54 H0	; Kapton bed heater thermistor
M305 P1 T100000 B4725 R4700 C7.06e-8 H0	; E3D V6 Semitec GT-104 thermistor cartridge

M570 S150				; Hot end may be a little slow to heat up so allow it 150 seconds. Recently uncommented

; Heater configurations
; Bed
M307 H0 B1				; Heater 0 (bed) use bang-bang control
; Heater 1 (Extruder)
M307 H1 A512.9 C267.0 D9.0 B0		; Heater 1 (extruder) use PID
M143 H1 S300 A1				; Heater overload protection

; Tool definitions
M563 P0 D0 H1 S"Extruder"   		; Define tool 0
G10 P0 S0 R0                        	; Set tool 0 operating and standby temperatures
M92 E780	                       	; Set extruder steps per mm.

; Z probe and compensation definition
;M558 P4 X0 Y0 Z0 H3 I1				; Z probe is a switch and is not used for homing any axes. I1 to invert for future.
M558 P4 I1 H3 F120 T6000 R0.4 S0.02 A5	 	; FSR JohnSL Z probe acts as a switch and is not used for axis homing (invert signal), Z probe dive height 3mm, probing speed 120mm/min, travel speed 6000mm/min, 0.4 sec recovery time, tolerance 0.02mm, max of 5 taps

; Z OFFSET ADJUSTMENT HERE
G31 X0 Y0 Z-0.28 P500				; lower # = more smooshed, higher # = less smooshed. IGNORE THE NEGATIVE SIGN.

; Define grid for mesh compensation
M557 R140 S20

; Enable emergency power response
M911 S21.0 R23.0 P"G91 M83 G1 Z20 E-4 F1000"

; select first hot end
T0					

; Load config override
M501

M122 output after only diving 350mm down from home:

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.02(RTOS) running on Duet WiFi 1.0 or 1.01
Board ID: 08DAM-999TL-MQ4S8-6J1F4-3SD6T-KN8MY
Used output buffers: 1 of 20 (15 max)
=== RTOS ===
Static ram: 25524
Dynamic ram: 98668 of which 0 recycled
Exception stack ram used: 408
Never used ram: 6472
Tasks: NETWORK(ready,544) HEAT(blocked,1232) MAIN(running,3812) IDLE(ready,200)
Owned mutexes:
=== Platform ===
Last reset 00:19:13 ago, cause: power up
Last software reset at 2019-01-17 17:19, reason: User, spinning module GCodes, available RAM 6480 bytes (slot 1)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 35.9, current 36.2, max 36.6
Supply voltage: min 24.0, current 24.3, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 0/248
Driver 1: standstill, SG min/max 0/1023
Driver 2: standstill, SG min/max 0/246
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2019-01-17 20:08:31
Cache data hit count 4294967295
Slowest loop: 4.15ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0
=== Move ===
Hiccups: 28, StepErrors: 0, LaErrors: 0, FreeDm: 240, MinFreeDm: 237, MaxWait: 8631ms, Underruns: 0, 0
Scheduled moves: 29, completed moves: 29
Bed compensation in use: none
Bed probe heights: 0.000 0.000 0.000 0.000 0.000
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Stack records: 2 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 202.66ms; fastest: 0.08ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
- WiFi -
Network state is running
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 4
WiFi firmware version 1.22
WiFi MAC address a0:20:a6:19:61:fb
WiFi Vcc 3.41, reset reason Turned on by main processor
WiFi flash size 4194304, free heap 29640
WiFi IP address 192.168.1.115
WiFi signal strength -54dBm, reconnections 0, sleep mode modem
Socket states: 0 0 0 0 0 0 0 0

Phaedrux

The hiccups are most certainly a result of too high microstepping. They may not always result in layer shifts though.

The Duet Wifi drivers only allow interpolation to 256x from a base x16 microstepping.

Try using x16 microstepping for all access with interpolation enabled. I think you'll find that the printer is far quieter, and I don't think you'll notice a difference in print quality from the lower microstepping value, other than the lack of hiccups and skipped layers.

Keep in mind you'll have to adjust your steps per mm, since it looks like you've got them set for the higher microstepping.

jacobf1

Have you done any significant changes to firmware or hardware recently? If the printer was working fine a week ago, something must have happened.

800 steps is correct for x64 microstepping and 16 teeth pulley.

garyd9

@mrsteve920

Is the problem the high number of hiccups or something else? I guess what I'm asking is if you were experiencing high hiccups before the other issues came up, or if you never checked before the other issues came up.

While printing, I easily see 10-20 hiccups every second on my own printer (also a ultibots d300vs+, also with microstepping set to 64 on the three delta towers with a 0.9 degree steppers.) I don't think the high hiccup rate is unusual for this printer. It's disturbing to see, and I also wonder if it's normal. Perhaps @dc42 could pop in and tell us if it's normal, something we should ignore, or something we should try to resolve.

Rather than going to back to 2.01, try going all the way back to firmware version 1.19 and check. (Does 1.19 report hiccups?) That's something that I've been meaning to try one day, but never seem to get around to it. (The reason for going all the way back to 1.19 is twofold: First, because it's what Brad at ultibots provides. Second, because version 1.19 is pre-RTOS...)

Anyway, I know you've said that you checked the belts, etc, but there still might be something not quite right. There's a script that David Crocker posted a while back that checks for backlash. It's posted here (https://forum.duet3d.com/topic/7934/smart-effector-consistency-query/3) and I also posted a copy of it on the ultibots facebook group a couple weeks ago. If you find that you're getting a lot of backlash, it might relate to your other symptoms.

dc42

Older firmware versions reported MaxReps instead of the hiccup count, but the cause was the same: the processor becoming overloaded because it had to generate steps at a very high rate. They don't necessarily cause a problem unless the hiccup rate gets extremely high. To reduce the hiccup count you will need to reduce the step pulse rate, which can most easily be done by reducing microstepping or by reducing travel speed.

deckingman

It seems odd though that the OP reported that the printer has always run at 64x without any problems until a week ago. According to my very quick "back of envelope" calculations, if we assume a maximum pulse frequency of 200,000 Hz, then 800 steps per mm should allow speeds of 250mm/sec. Of course, I know nothing about how Deltas work so maybe this is not a valid calculation.

dc42

@deckingman, on a delta when doing radial moves near the edges of the bed, the carriages move faster than the XY movement speed by a factor of up to about 3.

It's time I re-measured the maximum step rate of a Duet+RRF because there have been a lot of firmware changes since I did the original measurements. In particular, the firmware no longer does double/quad/octal stepping, because step-servo drives don't like it.

deckingman

@dc42 Thanks for the explanation - Delta's are very strange beasts which are beyond my comprehension

And yes, it would be good to know what the max step rate is now. Would you expect it to be higher or lower than 200kHz?

MrSteve920

@Phaedrux I can certainly try this for testing purposes and I'm sure it will work. I would be curious to know from others what the recommendation is for a delta. Is it better to run at x16 with interpolation or run at x32/x64 without?

@jacobf1 The only change that I did was that I got a new glass bed and installed a buildtak flex-plate system onto it. Of course I thought this might be the cause of my problems because the timing fits but after swapping the flex-plate system out for another just glass bed, my problems still exist. So either the flex-plate system caused a permanent change to the printer that I haven't been able to undo or it's something else. I analyzed the strength of the magnetic field under the glass bed, aluminum heat spreader, and kapton heater that I have on my bed and it was barely strong enough to hold a paperclip; the duet sits about 1.5in underneath this assembly. It is a 16 tooth pulley assembly as you indicated.

@garyd9 I'm assuming the problem is based on the number of hiccups because I've tried a lot other stuff to troubleshoot so far and this is all that I have left to go on; I never checked to see what number of hiccups the system was detecting before these issues started occuring. That's fascinating that your d300vs is also experiencing a high number of hiccups per second. I'll look into rolling back to the 1.19 firmware as you suggested and report back; as I recall going back that far requires more work than a usual update because of changes done in 1.19 and 1.20. I've been meaning to run that backlash script since it was posted on the facebook group but I haven't done so yet; I'll run it before trying any firmware changes.

@deckingman I concur that it is weird that I am only now experiencing these issues.

dc42

I've just done some tests on my delta. A step pulse rate of 120kHz to each of 3 motors moving simultaneously gives a small but acceptable hiccup rate (less than 50 over 200mm). If I increase it to 150kHz then the hiccup rate goes up by a factor of 10. So I recommend not more than 120kHz. When only 1 drive is moving, higher rates are possible - I'm guessing around 200kHz. On non-delta machines the maximum rates may be a little higher because the calculations are simpler.

This applies to stepper motors connected to the Duet's own drivers. For external stepper drivers the maximum rate will be lower because of the longer step pulse time and minimum step pulse interval. For machines using step-servo drives, the maximum step rate should be reduced further, because hiccups disrupt the smooth pulse train and that has a negative impact on the ability of a step-servo drive to predict what is coming next.

MrSteve920

@garyd9 I'm getting about 0.04mm difference in mean on that backlash test. That seems to indicate to me that I do have somewhat of a backlash issue, but I need some context to know how severe it is. This doesn't really surprise me since I just re-tensioned the belts and I probably went a little further than I should have. I know David's post in the thread that you mentioned had a difference in means of 0.009mm.

MrSteve920

@dc42 That explains my high number of hiccups since between the defaults for my config and the speeds I was requesting of the printer I would have needed a step pulse rate greater than 150kHz for pretty much all of my tests. I've put my config to x16 with interpolation and during basic calibrations I am producing no hiccups as expected. This was a good issue to get resolved, but I'm not convinced that this was the cause of my print issues since the print issues just started happening. I mentioned in a reply earlier that I recently installed a buildtak magnetic flexplate system onto my printer that I have since removed to do this testing. Can you think of anything that would be problematic about installing that system in the way I described? I was thinking about whether it could cause some level of emi, but that seems unlikely to me based on my tests and observations.

garyd9

@mrsteve920 said in High number of hiccups during basic moves:

I'll look into rolling back to the 1.19 firmware as you suggested and report back; as I recall going back that far requires more work than a usual update because of changes done in 1.19 and 1.20.

If/when you do that, according to dc42, the hiccups might be reported as MaxReps. PLEASE let us know what happens..

@dc42 said in High number of hiccups during basic moves:

I've just done some tests on my delta. A step pulse rate of 120kHz to each of 3 motors moving simultaneously gives a small but acceptable hiccup rate (less than 50 over 200mm). If I increase it to 150kHz then the hiccup rate goes up by a factor of 10. So I recommend not more than 120kHz.

To clarify: 120kHz to EACH of the motors for an overall rate of 360kHz?

While I realize that the nature of a delta's movement makes giving exact numbers difficult, could you please translate that to a suggested max steps/mm (M203) for X/Y/Z on a delta using 800 steps/mm?

(That leads to the question of if M203 parameters apply to the delta X/Y/Z carriage movement or to the X/Y/Z of cartesian movement. )

Thank you
Gary

dc42

@garyd9 said in High number of hiccups during basic moves:

To clarify: 120kHz to EACH of the motors for an overall rate of 360kHz?

Correct, although that doesn't mean you could drive a single motor at 360kHz without getting a high hiccup rate. Maybe 200kHz or a bit higher.

While I realize that the nature of a delta's movement makes giving exact numbers difficult, could you please translate that to a suggested max steps/mm (M203) for X/Y/Z on a delta using 800 steps/mm?

Only if you tell me the maximum travel speed that you want to use, and the angle between the rod and the horizontal when the effector is at the edge of the bed opposite the tower that the rod is attached to.

(That leads to the question of if M203 parameters apply to the delta X/Y/Z carriage movement or to the X/Y/Z of cartesian movement. )

To the Cartesian XYZ movement.

MrSteve920

@dc42 said in High number of hiccups during basic moves:

Only if you tell me the maximum travel speed that you want to use, and the angle between the rod and the horizontal when the effector is at the edge of the bed opposite the tower that the rod is attached to.

I got an angle of 14.5deg assuming his printer uses 375mm rods and using the delta radius from my last calibration with a 300mm diameter bed. Let's use a max travel speed of 175mm/s.

garyd9

@dc42 said in High number of hiccups during basic moves:

@garyd9 said in High number of hiccups during basic moves:

To clarify: 120kHz to EACH of the motors for an overall rate of 360kHz?

Correct, although that doesn't mean you could drive a single motor at 360kHz without getting a high hiccup rate. Maybe 200kHz or a bit higher.

...and "120kHz" translates to "120,000 steps (per motor) per second"?

Trying to put that into physical movement... With the driver configured for 800 steps per mm, that would result in 120kHz allowing 150mm/sec (for carriage movement.)

While I realize that the nature of a delta's movement makes giving exact numbers difficult, could you please translate that to a suggested max steps/mm (M203) for X/Y/Z on a delta using 800 steps/mm?

Only if you tell me the maximum travel speed that you want to use, and the angle between the rod and the horizontal when the effector is at the edge of the bed opposite the tower that the rod is attached to.

The max travel speed I ever use is around 120-150mm/sec, but max of 175 is also a good number just to get an idea.

I just measured 22.7 (+/- .2) degrees from horizontal (see image below.) It's possible that my "printable area" is smaller than @MrSteve920. This was after homing the printer and running "G1 Z250 Y-999."

If it matters, my delta arms are labeled as 375mm, but I'm using a slightly higher value. Here's the output of M665:

Diagonal 379.530, delta radius 208.700, homed height 411.472, bed radius 140.0, X 0.406°, Y 0.200°, Z 0.000°

Here's the image of the measurement
0_1547855185859_20190118_183857 - Copy.jpg

dc42

For a worst-case radial move close to the edge of the bed with the arm in that position, the ratio of carriage speed to XY speed is tan(90-22.7) = 2.4. So at 175mm/sec XY speed the carriage speed is 420mm/sec. Dividing 120kHz by 420mm/sec we get 285 microsteps/mm. So assuming you are using 16, 18 or 20 tooth 2mm pitch pulleys, if you are using 0.9deg motors then x16 microstepping with interpolation is the best choice. With 1.8deg motors you could use either x16 with interpolation, or x32.

garyd9

@dc42 said in High number of hiccups during basic moves:

For a worst-case radial move close to the edge of the bed with the arm in that position, the ratio of carriage speed to XY speed is tan(90-22.7) = 2.4. So at 175mm/sec XY speed the carriage speed is 420mm/sec. Dividing 120kHz by 420mm/sec we get 285 microsteps/mm. So assuming you are using 16, 18 or 20 tooth 2mm pitch pulleys, if you are using 0.9deg motors then x16 microstepping with interpolation is the best choice. With 1.8deg motors you could use either x16 with interpolation, or x32.

That's really interesting and I can plug different numbers into that to look at different scenarios. Thank you. Less speed gives more microsteps...

My only problem now is understanding what it actually means in terms of actual quality. I typically print 0.1mm layers, but I don't know if "hiccups" are really impacting quality or if they are just a statistic that I can ignore. As well, while I understand the mechanics of a "microstep", I'm not sure that I understand what they mean in terms of actual print quality. If I use 32 microsteps, that's 400 steps/mm (or 0.0025mm/step.) Multiplying the 2.4 ratio, it is still 0.006mm/step.

I'll take this to a new thread, as I think others might also be unsure of what some of this stuff actually means in terms of print quality... (https://forum.duet3d.com/topic/8688/stepper-movement-hiccups-and-microstepping)

MrSteve920

Thanks for the help everyone. It appears that changing my microstep values to something more appropriate for a delta with my components was the solution to my problems. I am now back to printing completely normal prints. I ended up using x16 with interpolation as suggested. It is still unclear to me as to why I only recently started having issues, but from this discussion it's clear that using x64 microstepping was not a recommended setting regardless of if I was having issues or not. Thanks again.

Phaedrux

@mrsteve920 do you notice any print quality change? Noise change?