23CL errors

kcorbedsellig

Since moving my project from my bench to a first prototype setup I'm running into errors and unresponsive 23CL's. As soon as I started running the 23CLs under load the first issues came up.

I started seeing errors in the console related to "short to vin". I remembered seeing this in the tuning manual and figured it would have something to do with tuning in the closed-loop plugin. The first issue I ran into here is that the 23CLs are configured as extruders and the plugin only allows tuning in axis. Since all 12 I have are under the exact same load I changed one to an unused axis and then autotuned it under load. I then took the PID and V and A values and added them to all my extruder drives like so:

M569.1 P82.0 T3 E2:4 R20 I2000 D0 A0 V1000

This did not fully resolve the issue but I seem to get the error less at this point.

I'm also seeing various CAN errors that I can't seem to find anything about in the documentation.

23/04/2024, 12:33:09	CAN response timeout: board 80, req type 6042, RID 58
23/04/2024, 12:33:08	CAN response timeout: board 80, req type 6043, RID 45
23/04/2024, 12:33:07	CAN response timeout: board 80, req type 6042, RID 34
23/04/2024, 12:33:06	CAN response timeout: board 80, req type 6018, RID 21

Besides that I also had one 23CL that is now completely unresponsive. I've replaced this one with my only spare. The replaced one does not work either but does stay responsive.
It could be a coincidence but could also be related that 23CL that is unresponsive was 81 and 80 gives errors, when placing 70 in the place of 80 the errors stay the same. Changing the order in the config does not change which 23cl has the error.

I've tried to take the load of the 23CL's and go back to the settings I had on my bench but that did not solve anything.

My setup is on the latest version of everything.

config:

; executed by the firmware on start-up
G4 S2                           ;wait for expansion boards to start

; General preferences
G91                             ; rel pos
M83                             ; ...but relative extruder moves
M550 P"conveyor"                ; set printer name
M669 K0                         ; printer mode

; Axis Limits
M208 X0 Y0 Z0 S1                                ; set axis minima
M208 X1000 Y99999 Z99999 S0        

; for pumps tuned settings
;M569.1 P71.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 71
;M569.1 P72.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 72
;M569.1 P73.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P74.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P75.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P76.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P77.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P78.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P79.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P80.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P70.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73
;M569.1 P82.0 T3 E2:4 R20 I2000 D0 A0 V1000 ; Configure the M23CL at CAN address 73

; for pumps untuned settings
M569.1 P71.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 71
M569.1 P72.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 72
M569.1 P73.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P74.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P75.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P76.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P77.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P78.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P79.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P80.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P70.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73
M569.1 P82.0 T3 E2:4 R100 I0 D0  ; Configure the M23CL at CAN address 73

M569 P71.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P72.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P73.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P74.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P75.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P76.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P77.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P78.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P79.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P80.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P70.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)
M569 P82.0 D4 S0 ; Set into closed-loop drive mode (D4) and reversed (S0)

; for conveyors
M569 P0.0 S1                                          ; physical drive 0.0 goes forwards
M569 P0.1 S1                                          ; physical drive 0.1 goes forwards
M350 X16
M566 X900 Y1300 E1300                                 ; set maximum instantaneous speed changes (mm/min)
M203 X90000.00 Y5000.00 E5000.00                      ; set maximum speeds (mm/min)
M201 X2000.00 Y1000.00 E1000.00                       ; set accelerations (mm/s^2)
M84 S30                                               ; Set idle timeout

M584 X0.0:0.1 E71.0:72.0:73.0:74.0:75.0:76.0:77.0:78.0:79.0:80.0:70.0:82.0
M906 X2000 Y2500 E2500:2500:2500:2500:2500:2500:2500:2500:2500:2500:2500:2500

M92 X100.00 Y80.00 E80:80:80:80:80:80:80:80:80:80:80:80 ; steps/mm

; extruder setup
M563 P0 D0:1:2:3:4:5:6:7:8:9:10:11 F0           
;M563 P0 D0:2:4:6:8:10:1:3:5:7:9:11 F0           
G10 P0 X0 Y0 Z0                                       ; set tool 0 axis offsets
G10 P0 R0 S0                                          ; set initial tool 0 active and standby temperatures to 0C
M568 P0 S1                                            ; enable mixing for tool 0
M567 P0 E1:1:1:1:1:1:1:1:1:1:1:1                      ; set mixing ratios for tool 0
T0

;sensors
M574 X1 S1 P"!io0.in"
M950 J1 C"!io1.in"                                     ; Input 4 uses io1.in pin, inverted
M581 P1 S0 T4 R0                                       ; Disable Steppers - trigger4.g

;no homing required to start
M564 H0

M122:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.5.1 (2024-04-22 13:55:46) running on Duet 3 MB6HC v1.01 (SBC mode)
Board ID: 08DJM-9P63L-DJ3T8-6JTDG-3SN6S-1V7RB
Used output buffers: 1 of 40 (27 max)
=== RTOS ===
Static ram: 155200
Dynamic ram: 86972 of which 0 recycled
Never used RAM 100964, free system stack 202 words
Tasks: SBC(2,ready,0.9%,372) HEAT(3,nWait 6,0.0%,357) Move(4,nWait 6,0.0%,336) CanReceiv(6,nWait 1,0.1%,797) CanSender(5,nWait 7,0.0%,334) CanClock(7,delaying,0.0%,350) TMC(4,nWait 6,9.8%,56) MAIN(2,running,83.1%,101) IDLE(0,ready,6.0%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:01:36 ago, cause: software
Last software reset at 2024-04-23 11:57, reason: User, FilamentSensors spinning, available RAM 100964, slot 0
Software reset code 0x600d HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
MCU temperature: min 31.0, current 31.1, max 33.1
Supply voltage: min 23.8, current 23.9, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0
Heap OK, handles allocated/used 99/7, heap memory allocated/used/recyclable 2048/100/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok, SG min 0, mspos 0, reads 5161, writes 14 timeouts 0
Driver 1: ok, SG min 0, mspos 0, reads 5162, writes 13 timeouts 0
Driver 2: ok, SG min 0, mspos 0, reads 5164, writes 11 timeouts 0
Driver 3: ok, SG min 9, mspos 9, reads 5164, writes 11 timeouts 0
Driver 4: ok, SG min 9, mspos 9, reads 5164, writes 11 timeouts 0
Driver 5: ok, SG min 9, mspos 9, reads 5164, writes 11 timeouts 0
Date/time: 2024-04-23 11:59:17
Slowest loop: 2.88ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0xfff00003
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 554, received 8547, lost 0, errs 0, boc 0
Longest wait 56ms for reply type 6041, peak Tx sync delay 64, free buffers 50 (min 49), ts 483/482/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 3472/3472
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x253c8
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server version 3.5.1 (2024-04-19 16:30:56, 32-bit)
HTTP+Executed:
> Executing M122
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 0
Full transfers per second: 38.49, max time between full transfers: 50.2ms, max pin wait times: 41.6ms/0.2ms
Codes per second: 0.73
Maximum length of RX/TX data transfers: 8184/1216

Any help is much appreciated as always.

kcorbedsellig

I've been doing some more testing and ran into more issues.

Each 23CL is connected to its own Meanwell 48V 2.5A power supply with a common ground and each 23CL faces the same load as they are all copies of each other.

For some reason, the 23CL with address 80 has been reporting shorts more than the others. I thought this had to do with tuning but that did not change much to the situation. I then noticed when measuring the voltage on the power supply of this 23CL that it was drawing 78V. I then took this supply out and tested it on my bench and found nothing wrong with it.

I also tested the other 23CLs in the system and they all worked fine when used individually including the one with address 80. However, upon using them as a single tool 80 started to report errors again and two others immediately stopped responding as well. I now have 3 23CLs out of 13 that don't show a blinking light anymore upon connecting. I haven't found a way to get these back to life as I can't connect them to reflash firmware or reset them. These for example also don't show as connected in the machine-specific information tab.

I have no clue what to test next and what could be the issue here. Any help would be greatly apreciated.

dc42

@kcorbedsellig please provide the response to sending command M122 B80 if the M23CL at that address is still connecting.

Regarding the M23CLs that don't blink in sync with the main board, what do the red and green LEDs on those motors do?

kcorbedsellig

@dc42

"M112 B80" for the 23CL that's showing most errors

M122 B80
Diagnostics for board 80:
Duet M23CL firmware version 3.5.1 (2024-04-19 14:42:52)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 45192, free system stack 186 words
Tasks: EncCal(1,nWait 6,0.0%,470) Move(3,nWait 7,0.0%,166) CLSend(3,nWait 6,0.0%,148) HEAT(2,nWait 6,0.1%,106) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,65) TMC(4,nWait 6,68.9%,326) MAIN(1,running,26.8%,248) IDLE(0,ready,0.0%,30) AIN(2,nWait 2,4.2%,255), total 100.0%
Owned mutexes:
Last reset 00:00:57 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, ok, SG min 0, mspos 8, reads 20377, writes 10329 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -8/3, peak Rx sync delay 16912, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 48.1, current 48.2, max 48.2
V12 voltage: min 12.1, current 12.1, max 12.1
MCU temperature: min 20.3C, current 22.4C, max 22.4C
Last sensors broadcast 0x00000000 found 0 154 ticks ago, 0 ordering errs, loop time 1
CAN messages queued 484, send timeouts 0, received 300, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Closed loop driver 0 mode: closed loop, pre-error threshold: 2.00, error threshold: 4.00, encoder type rotaryAS5047, position 826
Encoder reverse polarity: no, full rotations 0, last angle 825, minCorrection=-31.7, maxCorrection=35.1, agc 72, mag 4705, no error
Tuning mode: 0, tuning error: 0, collecting data: no
Control loop runtime (us): min=2, max=66, frequency (Hz): min=9259, max=14423

"M112 B73" for a 23CL that seems to work fine

M122 B73
Diagnostics for board 73:
Duet M23CL firmware version 3.5.1 (2024-04-19 14:42:52)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 45192, free system stack 186 words
Tasks: EncCal(1,nWait 6,0.0%,470) Move(3,nWait 7,0.0%,166) CLSend(3,nWait 6,0.0%,148) HEAT(2,nWait 6,0.1%,106) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,66) CanClock(5,nWait 1,0.0%,65) TMC(4,nWait 6,70.4%,326) MAIN(1,running,25.1%,273) IDLE(0,ready,0.0%,30) AIN(2,nWait 2,4.4%,255), total 100.0%
Owned mutexes:
Last reset 00:02:10 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, ok, SG min 0, mspos 8, reads 57763, writes 12606 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -8/2, peak Rx sync delay 189, resyncs 0/0, next timer interrupt due in 2 ticks, enabled, next step interrupt due in 4196851010 ticks, disabled
VIN voltage: min 48.2, current 48.2, max 48.3
V12 voltage: min 12.0, current 12.0, max 12.0
MCU temperature: min 22.5C, current 25.4C, max 25.4C
Last sensors broadcast 0x00000000 found 0 84 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 1073, send timeouts 0, received 668, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Closed loop driver 0 mode: closed loop, pre-error threshold: 2.00, error threshold: 4.00, encoder type rotaryAS5047, position -620
Encoder reverse polarity: no, full rotations -1, last angle 15764, minCorrection=-16.9, maxCorrection=20.9, agc 71, mag 4778, no error
Tuning mode: 0, tuning error: 0, collecting data: no
Control loop runtime (us): min=2, max=69, frequency (Hz): min=9036, max=14423

The ones that don't respond anymore don't show any LED's from power up to power down and don't respond to any console call.

Below the M115 response on a nonresponsive and a responsive 23CL. You get the same with a M122 for 23CL 72.0.

24/04/2024, 17:29:37	M115 B72
CAN response timeout: board 72, req type 6024, RID 77
24/04/2024, 17:29:29	M115 B80
Duet M23CL firmware version 3.5.1 (2024-04-19 14:42:52)

dc42

@kcorbedsellig regarding the motor at address 80 that still responds but is problematic, the reported encoder corrections are larger than expected (-31.7 and +35.1). Please remove the load from it, run encoder calibration, then run M122 B80 again and report the result. Note, encoder calibration should always be run with the motor unloaded.

Please email info@duet3d.com to discuss replacing the other motors. In the email, include a link to this thread and say where and when you purchased the M23CLs.

kcorbedsellig

@dc42 That's strange, I only ever ran the calibration without load. Is it required to rerun the calibration after a firmware upgrade? I'll recalibrate and report the results.

I'll get in touch with info@duet3d.com. Does that mean these 23CLs are bricked?

dc42

@kcorbedsellig it should not be necessary to re-run the calibration after a firmware upgrade. I'm interested to see whether that drive reports the same level of encoder corrections after recalibration.

Re the non-functioning 23CLs, I suspect that the internal fuse may have blown. It's soldered on the board, so not easily replaced by users. However, there is a firmware reload procedure that could be attempted. Do you have a multimeter?

dc42

@kcorbedsellig to run the firmware reload procedure:

Connect just one of the non-functioning motors to the CAN bus
Make sure you don't have any other devices at CAN address 123 on the CAN bus
Between the two connectors on the motor there is a reset button under a small hole in the casing. Find something pointed to press it with, such as a straightened-out paper clip. While holding the reset button down, power up the PSU supplying power to the motor. You can release the button a few seconds after powering it up. Then look for activity on the LEDs.

kcorbedsellig

@dc42 I'll run through these step first thing tomorrow morning. I have a multimeter, where might I find this fuse to check it?

kcorbedsellig

@dc42

Unfortunately pressing the reset button and powering on does not work on any of the unresponsive 23CLs. They don't give any sign of life. Both with and without a CAN cable connected while running the process.

For the recalibration of the one with address 80.0. I disconnected all other CAN devices and have run a single cable to a t-splitter of which one end is 120ohm resistors and the other is the 23CL. It does not want to calibrate and says it has no encoder configured.

M569.6 P80.0 V2
Error: M569.6: no encoder configured
25/04/2024, 11:12:08	M17
25/04/2024, 11:11:53	M115 B80
Duet M23CL firmware version 3.5.1 (2024-04-19 14:42:52)
25/04/2024, 11:07:55	Error: Expansion board 80 stopped sending status
25/04/2024, 11:07:14	M569.6 P80.0 V2
Error: M569.6: no encoder configured
25/04/2024, 11:06:59	M17
25/04/2024, 11:04:49	M115 B80
Duet M23CL firmware version 3.5.1 (2024-04-19 14:42:52)
25/04/2024, 11:04:00	M122 B80
Diagnostics for board 80:
Duet M23CL firmware version 3.5.1 (2024-04-19 14:42:52)
Bootloader ID: SAME5x bootloader version 2.8 (2023-07-25)
All averaging filters OK
Never used RAM 75108, free system stack 184 words
Tasks: Move(3,nWait 7,0.0%,175) CLSend(3,nWait 6,0.0%,148) HEAT(2,nWait 6,0.0%,131) CanAsync(5,nWait 4,0.0%,67) CanRecv(3,nWait 1,0.0%,80) CanClock(5,nWait 1,0.0%,65) TMC(2,nWait 6,23.5%,350) MAIN(1,running,73.9%,394) IDLE(0,ready,0.0%,40) AIN(2,nWait 2,2.5%,255), total 100.0%
Owned mutexes:
Last reset 00:07:03 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 48187, writes 13 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -8/5, peak Rx sync delay 184, resyncs 0/0, next timer interrupt due in 31 ticks, enabled, next step interrupt due in 3977492161 ticks, disabled
VIN voltage: min 48.1, current 48.1, max 48.2
V12 voltage: min 12.1, current 12.1, max 12.1
MCU temperature: min 17.3C, current 21.8C, max 21.8C
Last sensors broadcast 0x00000000 found 0 54 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 3410, send timeouts 0, received 2128, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Closed loop driver 0 mode: open loop, pre-error threshold: 2.00, error threshold: 4.00, encoder type none

For the possibly broken fuses on the unresponsive 23CLs. I've opened one and found what I believe must be the fuse. It's labeled F1 and can be found directly next to the power input. I've measured continuity on this fuse and it appears to not be broken.

dc42

@kcorbedsellig thanks. Regarding recalibrating the driver at address 80 it looks like no M569.1 command has been sent to that driver. Please send the correct M569.1 command and try again.