Duet3D 1HCL - Closed Loop Controller Beta test

[lirwin17]

@maxgyver

I've gone down a bit of a rabbit hole looking at why it can't run at top speeds - but I think I finally have an answer!

Forgive me, but I might talk about RPM here instead of mm/min. It just makes it a bit less printer-dependant.
For reference, RPM = (mm/min) * (step/mm) * (rev/step), and here are some common conversion that we've talked about (for an average printer):
22000 mm/min = 550 RPM
36000 mm/min = 900 RPM

In theory, the maximum speed is given by the PID loop frequency. Each iteration of the PID loop can command the motor to take 1 step (=rotate 0.005 degs). The PID loop currently runs at 12000 Hz.
This means that the theoretical max speed is 12000 x 0.005 x 60=3600 RPM.

Why then are we topping out at ~900 RPM (measured on my Ender 3, other's results may be different)
The answer is how long it takes to communicate the new motor currents to the driver ICs that we use.
At the minute, this takes ~0.064ms. At low RPMs, this is negligible. But at high RPMs, it means the motor has moved on a significant amount between deciding what current to set, and actually setting that current. At my max speed of 900RPM, it's moved on a whole 0.2 steps. We should be able to fix this by looking at the current RPM and 'predicting' how much the motor will have moved by the time the currents have been sent.

Thanks for pointing this out! It would be useful if I could gauge just how much of an issue this is. Would the current top speed be an issue for you? Or is it just a nice-to-have? If anyone else would have issues with this limitation, please do say
This feedback will then impact our decision on what to focus on improving next

[MaxGyver]

@lirwin17 said in Duet3D 1HCL - Closed Loop Controller Beta test:

Thanks for pointing this out! It would be useful if I could gauge just how much of an issue this is. Would the current top speed be an issue for you? Or is it just a nice-to-have? If anyone else would have issues with this limitation, please do say

For me, the whole point of switching to closed loop was to increase speed and reliability. In open loop my printer starts skipping steps at around 900 RPM (36000mm/min). During printing, I limit the travel speed to 24000 mm/s just to be on the safe side. So if it can reliably run at 900 RPM in closed loop, it would be okay for me. Everything more than that would be a nice to have.

[MaxGyver]

Okay, I have to revise my previous statement about the max. achievable speeds:

In fact my open loop travel speed is set to 600 mm/s (36000 mm/min or 900 RPM) and this already includes a 25% safety margin.

I just did some speed testing on my printer. I achieved speeds up to 1500 mm/s (2250 RPM) on a single axis without problems. I did not test the performance with multiple axis moving with other stuff like CoreXY interpolation and Z height adjustment going on. But it is an indicator of what the motor/driver is capable of in open loop.

In closed loop mode, my motor sound is kind of "wobbly", like it is slightly accelerating and decelerating on every turn. The "wobbly" sound becomes more noticeable with increasing speed. The Motor loses its position when the speed is over 400 mm/s (600 RPM).

Right now my setup is slower in closed loop than in open loop

-Max

[alankilian]

@maxgyver said in Duet3D 1HCL - Closed Loop Controller Beta test:

In closed loop mode, my motor sound is kind of "wobbly", like it is slightly accelerating and decelerating on every turn. The "wobbly" sound becomes more noticeable with increasing speed.

That sounds just like what I measured using a commercial external closed-loop stepper controller in THIS thread.

[CNCModeller]

@alankilian that language is consistent with my understanding too, although the encoder and reference material I use has lines = pulses, and counts = changes.

So a 600ppr encoder has 2400cpr.

Same principles all the same.

[supertb1]

@t3p3tony I have my firmware updated on both the 6HC and the tool board, I type in command M122 B50 and I get:
M122 B50
Diagnostics for board 50:
Duet EXP1HCL firmware version 3.4.0beta3 (2021-08-24 14:56:43)
Bootloader ID: SAME5x bootloader version 2.4beta (2021-07-23)
Never used RAM 52992, free system stack 2582 words
Tasks: Move(notifyWait,0.0%,126) HEAT(notifyWait,0.0%,108) CanAsync(notifyWait,0.0%,70) CanRecv(notifyWait,0.0%,79) CanClock(notifyWait,0.0%,72) TMC(notifyWait,15.0%,28) CLSend(notifyWait,0.0%,150) CLData(notifyWait,0.0%,142) MAIN(running,83.0%,437) IDLE(ready,0.0%,39) AIN(notifyWait,1.9%,265), total 100.0%
Last reset 00:12:21 ago, cause: software
Last software reset time unknown, reason: HardFault bfarValid precise, available RAM 51580, slot 0
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0x20030008 SP 0x2001eba8 Task MAIN Freestk 531 ok
Stack: 00000000 00000000 00000002 20030000 00030f39 0001b8f9 0001ea4c 41000000 bf800000 00000000 533c0605 04001200 a808a700 06420000 c3210828 19a21429 50708239 5d000040 2a60041c 5dd40122 4143051f 41200000 41c22c02 4142f6d6 20000010 0002699b 00000002
Closed loop enabled: yes, live status: 0x4, encoder type linearQuadrature, pre-error threshold: 20.000000, error threshold: 100.000000, coil A polarity: +, coil B polarity: -, tuning: 0, tuning error: 0x1f, position -1515870812, raw count = 65535, collecting data: no, ultimateGain=0.000000, oscillationPeriod=0.000000
Driver 0: position -50588, 106.5 steps/mm, standstill, reads 53855, writes 28 timeouts 0, SG min/max 0/455, steps req 62302 done 17087
Moves scheduled 5, completed 5, in progress 0, hiccups 0, step errors 0, maxPrep 25, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -4/5, peak Rx sync delay 180, resyncs 0/0, no step interrupt scheduled
VIN: 24.3V, V12: 12.2V
MCU temperature: min 27.4C, current 28.4C, max 28.4C
Ticks since heat task active 165, ADC conversions started 734738, completed 734737, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 170 ticks ago, loop time 0
CAN messages queued 5973, send timeouts 0, received 6695, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 292, adv 37084/37194

[supertb1]

@supertb1 ; Drives
G4 S1 ;wait for expansion boards to start
M569.1 P50.0 S1 T2 C20 E20 R100 I0 D0 ; Configure the 1HCL board at CAN address 50 with a quadrature encoder on the motor shaft that has 20 steps per motor full step.
M569.1 P51.0 S1 T2 C20 E20 R100 I0 D0 ; Configure the 1HCL board at CAN address 51 with a quadrature encoder on the motor shaft that has 20 steps per motor full step.
M569 P50.0 D4 S1 ; Configure the motor on the 1HCL at can address 50 as being in closed-loop drive mode (D4), Open loop (D2) and not reversed (S1) X axis
M569 P51.0 D2 S1 ; Configure the motor on the 1HCL at can address 51 as being in closed-loop drive mode (D4), Open loop (D2) and not reversed (S1) Y axis
M569 P0.0 S1 ; physical drive 0.0 goes forwards Z 1-2 axis
M569 P0.4 S1 ; physical drive 0.4 goes forwards Extruder
M584 X50.0 Y51.0 Z0.0:0.1 E0.4 ; set drive mapping
M671 X-122.25:689.20 Y0:0 S3.0 ; leadscrews at left (connected to Z0) and right (connected to Z1) of X axis
M350 X32 Y32 Z32 E16 I1 ; configure microstepping with interpolation
M92 X106.5 Y257 Z795.00 E339.00 ; set steps per mm
M566 X900.00 Y900.00 Z60.00 E120.00 ; set maximum instantaneous speed changes (mm/min)
M203 X6000.00 Y6000.00 Z180.00 E1200.00 ; set maximum speeds (mm/min)
M201 X500.00 Y500.00 Z20.00 E250.00 ; set accelerations (mm/s^2)
M906 X1000 Y3000 Z3000 E800 I30 ; set motor currents (mA) and motor idle factor in per cent
M84 S30 ; Set idle timeout
M917 X0 Y0 ; Set the closed loop axes to have a holding current of zero

[supertb1]

@supertb1 Product Model 42A03EC
Step Angle 1.8°±5% (full step no-load)
Rated Current 2.0 A
Holding Torque 0.3NM(42.5oz.in)
Encoder Resolution 4000ppr
Phase Resistance 1.6Ohm ± 10%
Phase Inductance 1.9mH ± 20%
Rotor Inertia 77g.cm²
Moto Size 424269mm
Shaft Diameter 8mm
Number of Leads 4
Description of Wire: Red (A+), Black (A-), Yellow(B+), Blue (B-)(new version);

[supertb1]

@supertb1 homex.g
; called to home the X axis
;
; generated by RepRapFirmware Configuration Tool v3.2.3 on Mon Mar 08 2021 08:53:31 GMT-0600 (Central Standard Time)
M569 P50.0 D0 ; Turn off closed loop

G91 ; relative positioning
G1 H2 Z5 F6000 ; lift Z relative to current position
G1 H1 X-240 F3000 ; move quickly to X axis endstop and stop there (first pass)
G1 H2 X5 F6000 ; go back a few mm
G1 H1 X-240 F240 ; move slowly to X axis endstop once more (second pass)

G90 ; absolute positioning
G1 X50 F3000 ; Move to a known-safe position
M400 ; Wait for the move to complete
G4 P500 ; Wait for the motor to settle
M569 P50.0 D4 ; Turn closed loop back on
M569.6 P50.0 V31 ; Perform the tuning manoeuvres for a quadrature encoder
G1 X0 ; Move back to X0

G1 H2 Z0 F6000 ; lower Z again

[supertb1]

@t3p3tony sorry about the multiple posts but it seems the posts are limited in size to 50 or 60 lines... is there anything else I can give you to make this easier?

[supertb1]

@t3p3tony I disconnect the motor and turn it one full revolution then M122 P50, resulting:
Diagnostics for board 50:
Duet EXP1HCL firmware version 3.4.0beta3 (2021-08-24 14:56:43)
Bootloader ID: SAME5x bootloader version 2.4beta (2021-07-23)
Never used RAM 52992, free system stack 2578 words
Tasks: Move(notifyWait,0.0%,126) HEAT(notifyWait,0.0%,108) CanAsync(notifyWait,0.0%,70) CanRecv(notifyWait,0.0%,79) CanClock(notifyWait,0.0%,72) TMC(notifyWait,15.5%,28) CLSend(notifyWait,0.0%,150) CLData(notifyWait,0.0%,142) MAIN(running,82.6%,380) IDLE(ready,0.0%,39) AIN(notifyWait,1.9%,265), total 100.0%
Last reset 00:36:45 ago, cause: software
Last software reset time unknown, reason: HardFault bfarValid precise, available RAM 51580, slot 0
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0x20030008 SP 0x2001eba8 Task MAIN Freestk 531 ok
Stack: 00000000 00000000 00000002 20030000 00030f39 0001b8f9 0001ea4c 41000000 bf800000 00000000 533c0605 04001200 a808a700 06420000 c3210828 19a21429 50708239 5d000040 2a60041c 5dd40122 4143051f 41200000 41c22c02 4142f6d6 20000010 0002699b 00000002
Closed loop enabled: yes, live status: 0x4, encoder type linearQuadrature, pre-error threshold: 20.000000, error threshold: 100.000000, coil A polarity: +, coil B polarity: -, tuning: 0, tuning error: 0x1f, position -1515866801, raw count = 4010, collecting data: no, ultimateGain=0.000000, oscillationPeriod=0.000000
Driver 0: position -50588, 106.5 steps/mm, standstill, reads 27092, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Moves scheduled 5, completed 5, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -5/4, peak Rx sync delay 182, resyncs 0/0, no step interrupt scheduled
VIN: 24.3V, V12: 12.2V
MCU temperature: min 27.4C, current 28.7C, max 28.9C
Ticks since heat task active 14, ADC conversions started 2187517, completed 2187516, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 18 ticks ago, loop time 0
CAN messages queued 11760, send timeouts 0, received 13191, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

[T3P3Tony]

@lirwin17 can you have a look at this please

[T3P3Tony]

@supertb1 please upgrade the 1HCL firmware to the latest here:
https://www.dropbox.com/sh/qkexrvhgsebbz0g/AAD9gfyS8UIU5Sau311WS1oLa?dl=0

[supertb1]

@t3p3tony I updated my firmware using the link above, E-stop, Ran
M569.6 P50.0 V31 then M122 B50:
M122 B50
Diagnostics for board 50:
Duet EXP1HCL firmware version 3.4.0beta3+1 (2021-09-14 15:07:31)
Bootloader ID: SAME5x bootloader version 2.4beta (2021-07-23)
Never used RAM 51580, free system stack 2582 words
Tasks: Move(notifyWait,0.0%,160) HEAT(notifyWait,0.0%,95) CanAsync(notifyWait,0.0%,70) CanRecv(notifyWait,0.0%,81) CanClock(notifyWait,0.0%,72) TMC(notifyWait,30.6%,358) CLSend(notifyWait,0.0%,148) CLData(notifyWait,0.0%,142) MAIN(running,67.4%,410) IDLE(ready,0.0%,39) AIN(notifyWait,2.0%,265), total 100.0%
Last reset 00:02:18 ago, cause: software
Last software reset time unknown, reason: HardFault bfarValid precise, available RAM 51580, slot 0
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0x20030008 SP 0x2001eba8 Task MAIN Freestk 531 ok
Stack: 00000000 00000000 00000002 20030000 00030f39 0001b8f9 0001ea4c 41000000 bf800000 00000000 533c0605 04001200 a808a700 06420000 c3210828 19a21429 50708239 5d000040 2a60041c 5dd40122 4143051f 41200000 41c22c02 4142f6d6 20000010 0002699b 00000002
Closed loop enabled: no, live status: 0, encoder type none, pre-error threshold: 0.000000, error threshold: 0.000000, reverse polarity: no, tuning: 0, tuning error: 0, collecting data: no, ultimateGain=0.000000, oscillationPeriod=0.000000, Control loop runtime (ms): min=0.000000, max=0.056000, avg=0.000000, Control loop frequency (Hz): min=5859.375000, max=18292.683594, avg=17441.861328
Driver 0: pos 0, 106.5 steps/mm, standstill, reads 56348, writes 19 timeouts 0, SG min/max 0/0, steps req 0 dMoves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -3/4, peak Rx sync delay 183, resyncs 0/0, no step interrupt scheduled
VIN: 24.3V, V12: 12.2V
MCU temperature: min 28.0C, current 28.0C, max 28.4C
Ticks since heat task active 200, ADC conversions started 136156, completed 136155, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 204 ticks ago, loop time 0
CAN messages queued 1154, send timeouts 0, received 1251, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
9/20/2021, 8:13:33 AM M569.6 P50.0 V31
Error: M569.6: Drive is not in closed loop mode.

as above in my config Channel 50 is in closed loop mode

[jay_s_uk]

@supertb1 can you increase the G4 S1 in your config to at least S2 if not S3?

[supertb1]

@jay_s_uk increased S1 to S3:
; Drives
G4 S3 ;wait for expansion boards to start
M569.1 P50.0 S1 T2 C20 E20 R100 I0 D0 ; Configure the 1HCL board at CAN address 50 with a quadrature encoder on the motor shaft that has 20 steps per motor full step.
M569.1 P51.0 S1 T2 C20 E20 R100 I0 D0 ; Configure the 1HCL board at CAN address 51 with a quadrature encoder on the motor shaft that has 20 steps per motor full step.
M569 P50.0 D4 S1 ; Configure the motor on the 1HCL at can address 50 as being in closed-loop drive mode (D4), Open loop (D2) and not reversed (S1) X axis
M569 P51.0 D2 S1 ; Configure the motor on the 1HCL at can address 51 as being in closed-loop drive mode (D4), Open loop (D2) and not reversed (S1) Y axis
M569 P0.0 S1 ; physical drive 0.0 goes forwards Z 1-2 axis
M569 P0.4 S1 ; physical drive 0.4 goes forwards Extruder
M584 X50.0 Y51.0 Z0.0:0.1 E0.4 ; set drive mapping
M671 X-122.25:689.20 Y0:0 S3.0 ; leadscrews at left (connected to Z0) and right (connected to Z1) of X axis
M350 X32 Y32 Z32 E16 I1 ; configure microstepping with interpolation
M92 X106.5 Y257 Z795.00 E339.00 ; set steps per mm
M566 X900.00 Y900.00 Z60.00 E120.00 ; set maximum instantaneous speed changes (mm/min)
M203 X6000.00 Y6000.00 Z180.00 E1200.00 ; set maximum speeds (mm/min)
M201 X500.00 Y500.00 Z20.00 E250.00 ; set accelerations (mm/s^2)
M906 X1000 Y3000 Z3000 E800 I30 ; set motor currents (mA) and motor idle factor in per cent
M84 S30 ; Set idle timeout
M917 X0 Y0 ; Set the closed loop axes to have a holding current of zero

[supertb1]

@jay_s_uk M122 B50
Diagnostics for board 50:
Duet EXP1HCL firmware version 3.4.0beta3+1 (2021-09-14 15:07:31)
Bootloader ID: SAME5x bootloader version 2.4beta (2021-07-23)
Never used RAM 51580, free system stack 2580 words
Tasks: Move(notifyWait,0.0%,160) HEAT(notifyWait,0.0%,108) CanAsync(notifyWait,0.0%,70) CanRecv(notifyWait,0.0%,81) CanClock(notifyWait,0.0%,72) TMC(notifyWait,30.6%,358) CLSend(notifyWait,0.0%,148) CLData(notifyWait,0.0%,142) MAIN(running,67.4%,400) IDLE(ready,0.0%,39) AIN(notifyWait,2.0%,265), total 100.0%
Last reset 00:01:06 ago, cause: software
Last software reset time unknown, reason: HardFault bfarValid precise, available RAM 51580, slot 0
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0x20030008 SP 0x2001eba8 Task MAIN Freestk 531 ok
Stack: 00000000 00000000 00000002 20030000 00030f39 0001b8f9 0001ea4c 41000000 bf800000 00000000 533c0605 04001200 a808a700 06420000 c3210828 19a21429 50708239 5d000040 2a60041c 5dd40122 4143051f 41200000 41c22c02 4142f6d6 20000010 0002699b 00000002
Closed loop enabled: no, live status: 0, encoder type none, pre-error threshold: 0.000000, error threshold: 0.000000, reverse polarity: no, tuning: 0, tuning error: 0, collecting data: no, ultimateGain=0.000000, oscillationPeriod=0.000000, Control loop runtime (ms): min=0.000000, max=0.057333, avg=0.001333, Control loop frequency (Hz): min=8241.758789, max=18750.000000, avg=17441.861328
Driver 0: pos 0, 106.5 steps/mm, standstill, reads 52300, writes 19 timeouts 0, SG min/max 0/0, steps req 0 dMoves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -4/4, peak Rx sync delay 177, resyncs 0/0, no step interrupt scheduled
VIN: 24.3V, V12: 12.2V
MCU temperature: min 27.8C, current 28.0C, max 28.0C
Ticks since heat task active 110, ADC conversions started 65762, completed 65761, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 115 ticks ago, loop time 0
CAN messages queued 579, send timeouts 0, received 603, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
9/20/2021, 8:50:40 AM M569.6 P50.0 V31
Error: M569.6: Drive is not in closed loop mode.

[supertb1]

@jay_s_uk No change

[jay_s_uk]

@supertb1 worth a shot

[lirwin17]

Hi @supertb1,

Sorry, I'm not 100% clear on the actual issue here. Thanks for providing your M122 output, from tuning error: 0x1f, it seems like no runtime tuning has happened. Similar to when an axis is un-homed, a driver that is untuned will not move, so if the issue is that the axis isn't moving then that would be the expected behaviour in this instance.
However, if you have run tuning, there is a deeper issue here.

Details of tuning can be found on this page. Specifically, the two 'what do I need to do?' sections (1, 2) are compulsory before the system will move (eventually these should be part of the online config tool, but not just yet for the beta test)

If you weren't aware of this page in the docs, that's really useful for us to know because it tells us we need to draw more attention to that page

Thanks!
Louis