[3.5.0-rc.1] 1HCL expansionboard needs resetting

[nikscha]

Hi David,
You probably noticed by now that I'm messing around with the 1HCl board and it's closed loop funcitonality. I noticed numerous times now that after a while (and especially after reconfiguring the different modes,) the board stops driving the stepper. It doesn't matter what mode. The board still responds to (and executes) commands sent via CAN.
First I just powercycled the printer to remedy this, now I'm using a macro:

M999 B21
M112

Note that I also need to reset the machine for this to work.

Duet 6HC + SBC

M122 here: https://pastebin.com/1nUX0N57
config here: https://pastebin.com/LihR1YwU

[dc42]

@nikscha please provide a M122 report for the 1HCL board after this issue has occurred and before you reset it.

[nikscha]

@dc42

I'm not sure how to reproduce this exactly, but by running config.g multiple times while changing modes inbetween triggered it.

M122 B21:

Diagnostics for board 21:
Duet EXP1HCL rev 1.0a or earlier firmware version 3.5.0-rc.1 (2023-08-31 16:24:13)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 46024, free system stack 178 words
Tasks: EncCal(1,nWait,0.0%,472) Move(3,nWait,0.0%,110) CLSend(3,nWait,0.0%,150) HEAT(2,nWait,0.1%,112) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,77) CanClock(5,nWait,0.0%,70) TMC(4,nWait,63.4%,325) MAIN(1,running,34.0%,251) IDLE(0,ready,0.0%,30) AIN(2,nWait,2.5%,265), total 100.0%
Last reset 01:04:50 ago, cause: software
Last software reset at 2023-09-06 09:52, reason: HardFault bfarValid precise, available RAM 45936, slot 2
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0xe3d48aa6 SP 0x2001a9a8 Task CLSe Freestk 161 ok
Stack: 2001aa14 e3d48aa6 295e0b16 2001aa13 20000b00 00027fdd 00034316 a1000000 20019ee4 00027fdd 20004da8 000270f5 00000000 11ab1500 00000040 0300a5a5 00000783 0000112e 458970d8 c3d59a88 c3d43bc2 402f63b3 7865c9a4 9a884589 3bc2c3d5 63b3c3d4 c983402f
Driver 0: pos 0, 396.0 steps/mm, ok, SG min 0, mspos 72, reads 23805, writes 6152 timeouts 0, steps req 0 done 3960
Moves scheduled 19, completed 19, in progress 0, hiccups 0, segs 6, step errors 0, maxPrep 151, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -8/6, peak Rx sync delay 12559, resyncs 0/0, next timer interrupt due in 8 ticks, enabled, next step interrupt due in 1422101038 ticks, disabled
VIN voltage: min 24.4, current 24.4, max 24.5
V12 voltage: min 12.0, current 12.1, max 12.1
MCU temperature: min 34.4C, current 36.2C, max 36.8C
Last sensors broadcast 0x00000000 found 0 173 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 31203, send timeouts 0, received 50547, lost 0, free buffers 38, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 376, adv 37054/37200
Closed loop driver 0 mode: closed loop, pre-error threshold: 0.00, error threshold: 0.00, encoder type rotaryAS5047, position -6751
Encoder reverse polarity: no, full rotations -1, last angle 9633, minCorrection=-33.4, maxCorrection=38.3, agc 52, mag 4755, no error
Tuning mode: 0, tuning error: 0, collecting data: no
Control loop runtime (us): min=4, max=57, frequency (Hz): min=9493, max=15957
Accelerometer: none
I2C bus errors 12, naks 0, contentions 0, other errors 0

[dc42]

@nikscha thanks for that. I'm a little puzzled because it says that the last reset cause was software, and the last software reset was on 6 September. Does the PC that runs DWC have the correct date set?

[nikscha]

@dc42 No, the PC running DWC has it's date set correctly.

I just sent an M999 B21 to reset the expansion board, and then queried another M122. The last reset is still 6th of September:

18/09/2023, 11:03:43  M122 B21
Diagnostics for board 21:
Duet EXP1HCL rev 1.0a or earlier firmware version 3.5.0-rc.1 (2023-08-31 16:24:13)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 76052, free system stack 202 words
Tasks: Move(3,nWait,0.0%,182) CLSend(3,nWait,0.0%,150) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,79) CanClock(5,nWait,0.0%,70) TMC(2,nWait,16.7%,359) MAIN(1,running,65.6%,391) IDLE(0,ready,16.0%,30) AIN(2,nWait,1.7%,265), total 100.0%
Last reset 00:00:04 ago, cause: software
Last software reset at 2023-09-06 09:52, reason: HardFault bfarValid precise, available RAM 45936, slot 2
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00430803 BFAR 0xe3d48aa6 SP 0x2001a9a8 Task CLSe Freestk 161 ok
Stack: 2001aa14 e3d48aa6 295e0b16 2001aa13 20000b00 00027fdd 00034316 a1000000 20019ee4 00027fdd 20004da8 000270f5 00000000 11ab1500 00000040 0300a5a5 00000783 0000112e 458970d8 c3d59a88 c3d43bc2 402f63b3 7865c9a4 9a884589 3bc2c3d5 63b3c3d4 c983402f
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min n/a, mspos 8, reads 47805, writes 12 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -1/1, peak Rx sync delay 177, resyncs 0/0, next timer interrupt due in 17 ticks, enabled, next step interrupt due in 4291533612 ticks, disabled
VIN voltage: min 24.4, current 24.4, max 24.5
V12 voltage: min 12.0, current 12.1, max 12.1
MCU temperature: min 31.2C, current 31.2C, max 31.2C
Last sensors broadcast 0x00000000 found 0 88 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 58, send timeouts 0, received 57, lost 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Closed loop driver 0 mode: open loop, pre-error threshold: 2.00, error threshold: 4.00, encoder type none
Accelerometer: none
I2C bus errors 12, naks 0, contentions 0, other errors 0

18/09/2023, 11:03:39 	Error: Expansion board 21 reconnected

18/09/2023, 11:03:38 	M999 B21  Board 21 resetting

Edit: This time the board was still responsive when I reset it

Edit2: I just checked, the 6th of September was when I uploaded custom firmware to the 1HCL for the first time (I only increased the margins for backlack in closed loop mode). As you can see from the M122 I "downgraded" the firmware again to the official release.

[dc42]

@nikscha ok so the crash recorded on 6 September was probably from your custom firmware.

[nikscha]

@dc42 Probably.
But the random "freezes" I'm experiencing shouldn't be related to that, as I'm now on the official release version, right?