Duet3D 1HCL - Closed Loop Controller Beta test

dc42

@supertb1 I think so, but @T3P3Tony will confirm.

supertb1

@dc42 well on the plus side the readings are not so jumbled up... looks like the step phase and raw encoder readings are 180 degrees out from the coil A current

lirwin17

@supertb1 Ah this is a much better graph - is this with C20?
Something else has caught my eye in your M122 - TMC(notifyWait,15.5%,26) - I'm pretty sure the TMC task runs at ~50% in the latest firmware, and around 15% in some older firmware. Is the firmware you're using definitely the 'unoffical' closed-loop dropbox release linked further up (https://www.dropbox.com/sh/qkexrvhgsebbz0g/AAD9gfyS8UIU5Sau311WS1oLa?dl=0)?

I think that because the 'unoffical' build has the same version number as the latest beta, any boards that were plugged in after you updated the firmware wouldn't update automatically - since it thinks it's already on the latest beta. @dc42 might be able to confirm if it works this way.
I assume the y-axis board is newly connected? if so, it might have fell into this trap.

Either way, I think it would be a good idea to at least try updating again to make sure you're on the 'unoffical' dropbox build linked above.

Just to be 100% sure, here's how I would update:

Download Duet3Firmware_EXP1HCL.bin from the link above
Drag and drop that bin file into the upload button on the system files tab
It should then ask if you want to install the firmware, and then go ahead and visibly install it - if it doesn't do this, this means there's a problem

Then after that's done, maybe try running M122 again, and see if that percentage on the TMC task has jumped from 15% to >40% - that would be a good sign that this was indeed the issue, and that it's now resolved.

If that's successful, you might want to set your C value back to 5, then please could you send another picture like that graph & we'll see if it's made any difference

supertb1

@lirwin17 Yes this is with C20, I will double check the firmware using this link you have provided. This is some good troubleshooting here... it is going to make it very easy for the end users to identify and solve issues with these kinds of encoded steppers.

supertb1

@lirwin17 I set the "C" value back to C5, performed home X and Y
M122 B50/51
9/30/2021, 12:30:25 PM M122 B51
Diagnostics for board 51:
Duet EXP1HCL firmware version 3.4.0beta3+1 (2021-09-14 15:07:31)
Bootloader ID: SAME5x bootloader version 2.4beta (2021-07-23)
Never used RAM 51304, free system stack 2571 words
Tasks: Move(notifyWait,0.0%,126) HEAT(notifyWait,0.0%,108) CanAsync(notifyWait,0.0%,70) CanRecv(notifyWait,0.0%,79) CanClock(notifyWait,0.0%,72) TMC(notifyWait,38.3%,344) CLSend(notifyWait,0.0%,148) CLData(notifyWait,0.0%,142) MAIN(running,59.6%,400) IDLE(ready,0.0%,39) AIN(notifyWait,2.0%,265), total 100.0%
Last reset 00:01:26 ago, cause: software
Last software reset at 2021-09-30 12:24, reason: HardFault bfarValid precise, available RAM 51580, slot 0
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0x20030008 SP 0x2001eba8 Task MAIN Freestk 531 ok
Stack: 00000000 00000000 00000001 20030000 00030f39 0001b8f9 0001ea4c 41000000 00000000 4072a200 e2080202 809d1f12 00884c02 45a0900e 81891780 0a003008 22000200 15220070 90840818 76001317 08124026 41200000 41c15176 41428ffe 20000010 0002699b 0000001a
Closed loop enabled: yes, live status: 0x4, encoder type rotaryQuadrature, pre-error threshold: 5.000000, error threshold: 10.000000, reverse polarity: yes, tuning: 0, tuning error: 0x1c, position 9, raw count = 0, collecting data: no, ultimateGain=0.000000, oscillationPeriod=0.000000, Control loop runtime (ms): min=0.000000, max=0.081333, avg=0.008000, Control loop frequency (Hz): min=6880.733887, max=18292.683594, avg=15625.000977
Driver 0: pos -122075, 257.0 steps/mm, standstill, reads 49188Moves scheduled 5, completed 5, in progress 0, hiccups 0, step errors 0, maxPrep 25, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -2/6, peak Rx sync delay 180, resyncs 0/0, no step interrupt scheduled
VIN: 24.2V, V12: 12.1V
MCU temperature: min 31.1C, current 31.3C, max 31.5C
Ticks since heat task active 23, ADC conversions started 85217, completed 85216, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 28 ticks ago, loop time 0
CAN messages queued 745, send timeouts 0, received 810, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 295, adv 37081/37189
9/30/2021, 12:30:13 PM M122 B50
Diagnostics for board 50:
Duet EXP1HCL firmware version 3.4.0beta3+1 (2021-09-14 15:07:31)
Bootloader ID: SAME5x bootloader version 2.4beta (2021-07-23)
Never used RAM 51304, free system stack 2575 words
Tasks: Move(notifyWait,0.0%,126) HEAT(notifyWait,0.0%,108) CanAsync(notifyWait,0.0%,70) CanRecv(notifyWait,0.0%,79) CanClock(notifyWait,0.0%,72) TMC(notifyWait,38.2%,344) CLSend(notifyWait,0.0%,148) CLData(notifyWait,0.0%,142) MAIN(running,59.8%,408) IDLE(ready,0.0%,39) AIN(notifyWait,2.0%,265), total 100.0%
Last reset 00:01:15 ago, cause: software
Last software reset time unknown, reason: HardFault bfarValid precise, available RAM 51580, slot 2
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0x20030008 SP 0x2001eba8 Task MAIN Freestk 531 ok
Stack: 00000000 00000000 00000002 20030000 00030f39 0001b8f9 0001ea4c 41000000 bf800000 00000000 533c0605 04001200 a8888700 06420000 c2010829 19a21029 50708239 5d000040 2a60041c 54540122 04801201 41200000 41c27924 4142a3fe 20000010 0002699b 00000002
Closed loop enabled: yes, live status: 0x4, encoder type rotaryQuadrature, pre-error threshold: 5.000000, error threshold: 10.000000, reverse polarity: no, tuning: 0, tuning error: 0x1c, position 1, raw count = 0, collecting data: no, ultimateGain=0.000000, oscillationPeriod=0.000000, Control loop runtime (ms): min=0.000000, max=0.069333, avg=0.006667, Control loop frequency (Hz): min=6756.756836, max=18292.683594, avg=15625.000977
Driver 0: pos -50588, 106.5 steps/mm, standstill, reads 50380, Moves scheduled 5, completed 5, in progress 0, hiccups 0, step errors 0, maxPrep 25, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -3/3, peak Rx sync delay 180, resyncs 0/0, no step interrupt scheduled
VIN: 24.3V, V12: 12.2V
MCU temperature: min 29.3C, current 29.5C, max 29.5C
Ticks since heat task active 63, ADC conversions started 73919, completed 73918, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 67 ticks ago, loop time 0
CAN messages queued 653, send timeouts 0, received 707, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 297, adv 37078/37193

M569.1 P51.0 T2 C5 E5:10

51_2021-09-30_12.32.37 C5Yaxis.csv

M569.1 P51.0 T2 C20 E5:10
51_2021-09-30_12.40.33 C20Yaxis.csv
so I think whatever changes this latest firmware has done has helped with the C5 value

lirwin17

@supertb1 That's worked

So from those graphs it's definitely C5, so keep it at that
From your M122: tuning error: 0x1c, this is yet to be documented, but this happens when not all the required tuning moves are made. From the docs:

manoeuvres 1, 2, 4, 8 and 16 are required

And 0x1c means 4, 8 & 16 haven't (yet) been run - so just try to home (assuming your homing file includes running those moves - which it should if you followed the tuning page - i.e. you should have an M569.6 ... V31 in your homing file somewhere)

Let me know if your motor then holds position after completing homing

If that works, we can move onto the x axis. Perhaps try unplugging your y motor and encoder from board 51 and plugging your x motor and encoder into board 51. I would expect your x axis to then just work (albeit it will think it's Y). If it doesn't we'll need to investigate your motor

supertb1

@lirwin17 so I did what you said and the problem follows from one card to the next with the X axis motor, on V31 maneuver it returns a G28 reverse polarity error. That said the Y axis is now holding position but it is not moving back to 0 in absolute following the suggested homing sequence:
; homey.g
; called to home the Y axis
;
; generated by RepRapFirmware Configuration Tool v3.2.3 on Mon Mar 08 2021 08:53:32 GMT-0600 (Central Standard Time)
M569 P51.0 D0 ; Turn off closed loop

G91 ; relative positioning
G1 H2 Z5 F6000 ; lift Z relative to current position
G1 H1 Y-240 F3000 ; move quickly to X axis endstop and stop there (first pass)
G1 H2 Y5 F6000 ; go back a few mm
G1 H1 Y-240 F240 ; move slowly to X axis endstop once more (second pass)

G90 ; absolute positioning
G1 Y25 F3000 ; Move to a known-safe position
M400 ; Wait for the move to complete
G4 P500 ; Wait for the motor to settle
M569 P51.0 D4 ; Turn closed loop back on
M569.6 P51.0 V31 ; Perform the tuning manoeuvres for a quadrature encoder
G1 Y0 ; Move back to Y0

G1 H2 Z0 F6000 ; lower Z again

51_2021-09-30_17.48.08 C5 Y axis.csv

So i have a sneaking suspicion that the problem may lie in the extension joint made on the encoder wires that I am going to look into and get back to you on...How do I get the now closed loop Y axis to move back to the Y axis limit switch or within 10mm of it?

9/30/2021, 5:43:25 PM M122 B51
Diagnostics for board 51:
Duet EXP1HCL firmware version 3.4.0beta3+1 (2021-09-14 15:07:31)
Bootloader ID: SAME5x bootloader version 2.4beta (2021-07-23)
Never used RAM 51304, free system stack 2575 words
Tasks: Move(notifyWait,0.0%,126) HEAT(notifyWait,0.0%,108) CanAsync(notifyWait,0.0%,70) CanRecv(notifyWait,0.0%,79) CanClock(notifyWait,0.0%,72) TMC(notifyWait,40.1%,344) CLSend(notifyWait,0.0%,148) CLData(notifyWait,0.0%,142) MAIN(running,57.8%,380) IDLE(ready,0.0%,39) AIN(notifyWait,2.0%,265), total 100.0%
Last reset 00:02:00 ago, cause: software
Last software reset at 2021-09-30 12:24, reason: HardFault bfarValid precise, available RAM 51580, slot 0
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0x20030008 SP 0x2001eba8 Task MAIN Freestk 531 ok
Stack: 00000000 00000000 00000001 20030000 00030f39 0001b8f9 0001ea4c 41000000 00000000 4072a200 e2080202 809d1f12 00884c02 45a0900e 81891780 0a003008 22000200 15220070 90840818 76001317 08124026 41200000 41c15176 41428ffe 20000010 0002699b 0000001a
Closed loop enabled: yes, live status: 0, encoder type rotaryQuadrature, pre-error threshold: 5.000000, error threshold: 10.000000, reverse polarity: yes, tuning: 0, tuning error: 0, position -7870, raw count = 10302, collecting data: no, ultimateGain=0.000000, oscillationPeriod=0.000000, Control loop runtime (ms): min=0.005333, max=0.089333, avg=0.013333, Control loop frequency (Hz): min=6818.182129, max=16666.667969, avg=14423.078125
Driver 0: pos -122075, 257.0 steps/mm, standstill, reads 34Moves scheduled 5, completed 5, in progress 0, hiccups 0, step errors 0, maxPrep 22, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -2/4, peak Rx sync delay 180, resyncs 0/0, no step interrupt scheduled
VIN: 24.2V, V12: 12.1V
MCU temperature: min 30.2C, current 30.9C, max 30.9C
Ticks since heat task active 199, ADC conversions started 118514, completed 118513, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 204 ticks ago, loop time 0
CAN messages queued 318, send timeouts 0, received 329, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 297, adv 37079/37188

supertb1

@lirwin17 I started looking into the possible issue with the splicing/ solder joint on my X axis encoder wires... I couldn't find any real issue there and I decided to to take a stab at it and switch the A+/B+ wires, low and behold it actually passed the V31 maneuver and holds position.... I couldn't believe it... the 1HCL board was trying to tell me the whole time what was wrong...lol. success... now all I need is for the homing sequence to actually return both axis to the near limit switch end of the travel and we will be good to go.

9/30/2021, 7:05:25 PM M122 B50
Diagnostics for board 50:
Duet EXP1HCL firmware version 3.4.0beta3+1 (2021-09-14 15:07:31)
Bootloader ID: SAME5x bootloader version 2.4beta (2021-07-23)
Never used RAM 51304, free system stack 2575 words
Tasks: Move(notifyWait,0.0%,126) HEAT(notifyWait,0.0%,95) CanAsync(notifyWait,0.0%,70) CanRecv(notifyWait,0.0%,79) CanClock(notifyWait,0.0%,72) TMC(notifyWait,43.1%,344) CLSend(notifyWait,0.0%,148) CLData(notifyWait,0.0%,142) MAIN(running,54.9%,418) IDLE(ready,0.0%,39) AIN(notifyWait,1.9%,265), total 100.0%
Last reset 00:17:36 ago, cause: power up
Last software reset time unknown, reason: HardFault bfarValid precise, available RAM 51580, slot 2
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0x20030008 SP 0x2001eba8 Task MAIN Freestk 531 ok
Stack: 00000000 00000000 00000002 20030000 00030f39 0001b8f9 0001ea4c 41000000 bf800000 00000000 533c0605 04001200 a8888700 06420000 c2010829 19a21029 50708239 5d000040 2a60041c 54540122 04801201 41200000 41c27924 4142a3fe 20000010 0002699b 00000002
Closed loop enabled: yes, live status: 0, encoder type rotaryQuadrature, pre-error threshold: 5.000000, error threshold: 10.000000, reverse polarity: yes, tuning: 0, tuning error: 0, position -6528, raw count = 58926, collecting data: no, ultimateGain=0.000000, oscillationPeriod=0.000000, Control loop runtime (ms): min=0.000000, max=0.096000, avg=0.013333, Control loop frequency (Hz): min=6465.517578, max=18292.683594, avg=14423.078125
Driver 0: pos -50588, 106.5 steps/mm, standstill, reads 567Moves scheduled 5, completed 5, in progress 0, hiccups 0, step errors 0, maxPrep 25, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -4/5, peak Rx sync delay 183, resyncs 0/0, no step interrupt scheduled
VIN: 24.3V, V12: 12.2V
MCU temperature: min 26.8C, current 28.4C, max 28.9C
Ticks since heat task active 19, ADC conversions started 1034352, completed 1034351, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 23 ticks ago, loop time 0
CAN messages queued 8499, send timeouts 0, received 9537, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 296, adv 37083/37183

; homex.g
; called to home the X axis
;
; generated by RepRapFirmware Configuration Tool v3.2.3 on Mon Mar 08 2021 08:53:31 GMT-0600 (Central Standard Time)
M569 P50.0 D0 ; Turn off closed loop

G91 ; relative positioning
G1 H2 Z5 F6000 ; lift Z relative to current position
G1 H1 X-240 F3000 ; move quickly to X axis endstop and stop there (first pass)
G1 H2 X5 F6000 ; go back a few mm
G1 H1 X-240 F240 ; move slowly to X axis endstop once more (second pass)

G90 ; absolute positioning
G1 X50 F3000 ; Move to a known-safe position
M400 ; Wait for the move to complete
G4 P500 ; Wait for the motor to settle
M569 P50.0 D4 ; Turn closed loop back on
M569.6 P50.0 V31 ; Perform the tuning manoeuvres for a quadrature encoder
G1 X0 ; Move back to X0

G1 H2 Z0 F6000 ; lower Z again

50_2021-09-30_19.06.20 C5 X axis.csv

Drive Config:
G4 S1 ;wait for expansion boards to start
M569.1 P50.0 T2 C5 E5:10 ; Configure the 1HCL board at CAN address 50 with a quadrature encoder on the motor shaft that has 20 steps per motor full step.
M569.1 P51.0 T2 C5 E5:10 ; Configure the 1HCL board at CAN address 51 with a quadrature encoder on the motor shaft that has 20 steps per motor full step.
M569 P50.0 D4 S1 ; Configure the motor on the 1HCL at can address 50 as being in closed-loop drive mode (D4), Open loop (D2) and not reversed (S1) X axis
M569 P51.0 D4 S1 ; Configure the motor on the 1HCL at can address 51 as being in closed-loop drive mode (D4), Open loop (D2) and not reversed (S1) Y axis
M569 P0.0 S1 ; physical drive 0.0 goes forwards Z 1-2 axis
M569 P0.4 S1 ; physical drive 0.4 goes forwards Extruder
M584 X50.0 Y51.0 Z0.0:0.1 E0.4 ; set drive mapping
M671 X-122.25:689.20 Y0:0 S3.0 ; leadscrews at left (connected to Z0) and right (connected to Z1) of X axis
M350 X16 Y16 Z16 E16 I1 ; configure microstepping with interpolation
M92 X106.5 Y257 Z795.00 E339.00 ; set steps per mm
M566 X900.00 Y900.00 Z60.00 E120.00 ; set maximum instantaneous speed changes (mm/min)
M203 X6000.00 Y6000.00 Z180.00 E1200.00 ; set maximum speeds (mm/min)
M201 X500.00 Y500.00 Z20.00 E250.00 ; set accelerations (mm/s^2)
M906 X1000 Y3000 Z3000 E800 I30 ; set motor currents (mA) and motor idle factor in per cent
M84 S30 ; Set idle timeout
M917 X0 Y0 ; Set the closed loop axes to have a holding current of zero

supertb1

@lirwin17 today I descided to do some more maneuvers... IE home the axis, move the axis around on it's travel and I found something strange... the axis homes fine, completes the V31 maneuver and stops around the safe distance traveled. Then when I tell it to move in a positive direction the drive makes the stepper move in the opposite direction intended.... lol... one step forward... two steps back...lol.. and the next question what should a homeall.g look like

supertb1

@lirwin17 Good afternoon Louis, today I decided to put it back in open loop and try a test print, after all the changes the first thing I wanted to do is a height map and a bed leveling... I put my microsteps back to 32... now that I got that part right.. "facepalm and long sigh" I put the B+ wires on the A+ terminal and the A+ terminals on the B+ terminal and did a homing maneuver on both axis with good results.. the axis is moving back to the home position and moving in the correct direction. Finally getting this going.

50_2021-10-04_10.11.00 X axis manuever.csv

51_2021-10-04_10.13.33 Y axis manuever.csv I'm going to mess with the home all to try and get everything working correctly... fingers crossed
So here is my Home all config currently:

G91 ; relative positioning
G1 H2 Z5 F6000 ; lift Z relative to current position
G1 H1 X-605 Y-605 F1800 ; move quickly to X and Y axis endstops and stop there (first pass)
G1 H2 X5 Y5 F6000 ; go back a few mm
G1 H1 X-605 Y-605 F360 ; move slowly to X and Y axis endstops once more (second pass)
G90 ; absolute positioning
G1 X50 Y50 F3000 ; Move to a known-safe position
M400 ; Wait for the move to complete
G4 P500 ; Wait for the motor to settle
M569 P50.0 D4 ; Turn closed loop back on X axis
M569 P51.0 D4 ; Turn closed loop back on Y axis
M569.6 P50.0 V31 ; Perform the tuning manoeuvres for a quadrature encoder X axis
M569.6 D51.0 V31 ; Perform the tuning manoeuvres for a quadrature encoder Y axis
G1 X0 Y0 ; Move back to X0
G1 X10 Y10 F6000 ; go to first bed probe point and home Z
G30 ; home Z by probing the bed
This does not work.... it just sits there. any advice here woud be great... making great headway

supertb1

@dc42 I'm well into my first print in closed loop and all is well, I played with that tuning procedure a bit and after I had tried a few times to enter the PDand I variables according to the examples, the X axis it wouldn't perform the homing maneuver so I just put it back the way it was and I am happy with the amount of noise, accuracy, and temp of the motors during operations... so all in all a success. Looking forward to the next update, advice on homeall.g and any streamlining/improvements that have been made. let me know if there is any procedures you would like me to try.

dc42

@supertb1 we hope to release firmware 3.4beta5 shortly, which includes some improvements to the closed loop tuning mechanism. For details, see https://github.com/Duet3D/RepRapFirmware/wiki/Changelog-RRF-3.x-Beta-&-RC#reprapfirmware-340beta5.

supertb1

@dc42 Just an update, I have beta 5 installed and am in to my 3rd three day print and there is no sign of any missed steps or shifting of any kind, my prints are holding a tight tolerance and I am very happy with the results... how long before I can buy another set of boards?

T3P3Tony

@supertb1 thanks for the feedback! We will have information about the production run once the shcedule is confirmed.

MaxGyver

I would like to give a quick thank you to everyone involved with the development of the 1HCL-Boards and the closed loop plugin.

Running my printer in closed loop is a real quantum leap for me. I can now comfortably print at speeds and accelerations that were really pushing the limits in open loop. In addition, the new level of reliability enables me to run larger print jobs for longer unsupervised periods. My printer was already very tough and reliable, but adding closed loop control fixed the last weak link in the chain.

I am eagerly awaiting further developments and improvements. For example, having linear encoders that measure the head's position on the axis independent of belts and pulleys would be the ultimate closed loop solution. But I am digressing.

Good job everyone!

Cheers Max

A Former User

@dc42
Hey
While you're printing with this pieces of gold, I ask myself where I can buy this board or how I can also test it?

I'm producing custom extra large toolchanger, so it would be that helpful!!
Cheers
Richard

dc42

@maxgyver what kinematics does your machine use? We've experienced some issues converting one of our CoreXY machines to use the 1HCL board and motors+encoders; and we don't know whether it is as issues with this particular machine+motors+encoders or a more general problem with closed loop control of coupled motors.

A Former User

@dc42
I have cartesian printers.
4 y motors
2 y motors
1x motor

X and Y working with a closed loop system would be nice.

Cheers
Richard

samlogan87

@T3P3Tony is there any news on these boards? Still keen to get 2

MaxGyver

@dc42 My Kinematics are IDEX (Dual Markforged aka Haq XY) with dual Y-Motors.

; Kinematics
M669 K11 Y1:1:0:-1	; Matrix mapping for Axis - X:Y:Z:U

So far, I had no mayor issues converting to closed loop.