[3.5b1?] CAN from 6HC to Mini 5+ fails after board reset

[Diamondback]

So i have a Duet 3 6HC as my CAN master and a Duet 3 Mini 5+ connected via CAN in expansion mode. Things generally work fine, however, when DWC asks me to restart the master board after changes to config.g something seems to break with regards to the CAN connection.
On the Mini 5 the Status LED starts flashing very rapidly and the CAN connection breaks down.

3.5b1 is the first firmware version I use on this setup, so can't tell if it's an older issue or new (I know, I know, new hardware and then immediately beta firmware... Don't shoot me, I really wanted the custom array support )

[Phaedrux]

Can you share your config.g and the results of M122 for each board?

[Diamondback]

@Phaedrux
config.g (let me know if you need the macros as well)

;M929 S3                                        ;enable debug logging
;M575 P1 S1 B57600                              ;Enable PanelDue Port
G90                                             ; send absolute coordinates...
M83                                             ; ...but relative extruder moves
M550 P"V-Core Toolchanger"                      ; set printer name

; Enable network
M552 P0.0.0.0 S1

M669 K1                                         ; select CoreXY mode

G4 S2 ; wait for expansion boards to start

; Drives
;XY1: back left
;XY2: back right
M569 P0.2 S0                                      ; stepper XY1
M569 P0.3 S0                                      ; stepper XY2
M569 P0.4 S1                                      ; stepper Coupler
M569 P100.0 S0                                    ; stepper E0
M569 P100.1 S0                                    ; stepper E1
M569 P100.2 S0                                    ; stepper E2
M569 P100.3 S0                                    ; stepper E3
M569 P0.0 S0                                      ; stepper Z1
M569 P0.1 S0                                      ; stepper Z2
M569 P0.5 S0                                      ; stepper Z3

M584 X0.3 Y0.2 C0.4                               ; set drive mapping (motion system X Y C)
M584 E100.0:100.1:100.2:100.3                     ; set drive mapping (extruders)
M584 Z0.0:0.1:0.5                                 ; set drive mapping (motion system Z)
;Leadscrew order (starting back center, then clockwise):
;1: back center
;2: front right
;3: front left
M671 X-12:216:-203 Y222:-179:-179 S30             ; set Z leadscrew positions


; Axis Limits
if !exists(global.hasRunInit)
	global minY = -266
	global maxY = 115

M208 X-148:160 Y{global.minY,global.maxY}       ; set axis minima & maxima
M208 C0:250 Z-2:270                             ; set axis minima & maxima

M92 X80.00 Y80.00                               ; X/Y: 1.8°/step
;M92 X160.00 Y160.00                            ; X/Y: 0.9°/step
M92 C200.00                                     ; C: E3D coupler
M92 Z800.00                                     ; Z: TR8x4 leadscrews
M350 X16 Y16 I1                                 ; configure microstepping with interpolation
M350 C16 Z16 I1                                 ; configure microstepping with interpolation
M566 X600 Y600 C2 Z70                           ; set maximum instantaneous speed changes (mm/min)
M203 X30000 Y30000 C10000 Z2000                 ; set maximum speeds (mm/min)
M201 X5000 Y5000 C500 Z500                      ; set accelerations (mm/s^2) was: 2k
M204 P600 T5000
;M204 P5000 T5000

M906 X2100 Y2100 I50                            ; set motor currents (mA) and motor idle factor in per cent
M906 C500 I30                                   ; set motor currents (mA) and motor idle factor in per cent
M906 Z1800 I30                                  ; set motor currents (mA) and motor idle factor in per cent
M906 E500:500:500:500 I30                       ; set motor currents (mA) and motor idle factor in per cent (2x Flex3Drive 2x LGX)
M84 S30                                         ; Set idle timeout    

;=================== Extruder ======================
M92 E408:408:408:408                            ; set steps per mm 
M350 E16:16:16:16 I1                            ; configure microstepping 
M566 E500:500:500:500                           ; set maximum instantaneous speed changes (mm/min) ; used to be at 1200
M203 E15000:15000:15000:15000                   ; set maximum speeds (mm/min) 
M201 E3000:3000:3000:3000                       ; set accelerations (mm/s^2)
;====================================================

; Endstops
M574 X2 S1 P"0.io0.in"                          ; configure active-high endstop for high end on X via pin xstop
M574 Y1 S1 P"0.io1.in"                          ; configure active-high endstop for high end on Y via pin ystop
M574 Z0                                         ; No Z-Endstop (use as probe instead)

; Z-Probe
if !exists(global.hasRunInit)
	global zProbeSpeedFeed = 250
	global zProbeSpeedFeedLevel = 2100
	global zProbeSpeedTravel = 20000
	global zProbeDiveHeightLevel = 30
	global zProbeDiveHeightMesh = 3
	global zProbePin = "0.io2.in"
	global zProbeType = 8
	global zProbeTriggerValue = 200
	global minXMesh = -146 
	global maxXMesh = 150
	global minYMesh = -150
	global maxYMesh = 150
	global meshPointSpacing = 30

M558 K0 P{global.zProbeType} C{global.zProbePin} H{global.zProbeDiveHeightMesh} F{global.zProbeSpeedFeed} T{global.zProbeSpeedTravel}  ; default Z probe with slow speed
;M558 K0 P8 C"0.io2.in" H3 F250 T20000  ; default Z probe with slow speed
;M558 K1 P8 C"0.io2.in" H30 F2100 T20000  ; high speed Z probe for initial leveling and homing runs
M558 K2 P8 C"0.io4.in" H10 F100 T20000  ; set Z probe type to unfiltered switch and the dive height + speeds

G31 K0 X0 Y0 Z0 P200 
;G31 K1 X0 Y0 Z0 P200
G31 K2 X0 Y0 Z0 P200 

;G31 K0 X0 Y0 Z0 P{global.zProbeTriggerValue}                               ; set Z probe trigger value, offset and trigger height, offsets here are hardcoded in leveling macro as well

M557 X{global.minXMesh, global.maxXMesh} Y{global.minYMesh, global.maxYMesh} S{global.meshPointSpacing}   ; define mesh grid
M376 H30                                        ; Set bed correction taper

; Heaters
M308 S0 P"100.temp0" Y"thermistor" T100000 B3950           ; configure sensor 0 as thermistor 
M950 H0 C"100.out0" T0 Q10                                 ; create bed heater output and map it to sensor 0
M143 H0 S121                                               ; set temperature limit for heater 0 to 121C
M307 H0 B0 R0.676 C345.1 D2.68 S1.00 V24.3                 ; Bed PID Tuning
M140 H0                                                    ; map heated bed to heater 0

M308 S1 P"0.temp1" Y"thermistor" T100000 B4725 C7.06e-8    ; configure sensor 1 as thermistor
M950 H1 C"0.out1" T1                                       ; create nozzle heater output and map it to sensor 1
M143 H1 S301                                               ; set temperature limit for heater 1 to 301C

M308 S2 P"0.temp2" Y"thermistor" T100000 B4725 C7.06e-8    ; configure sensor 2 as thermistor
M950 H2 C"0.out2" T2                                       ; create nozzle heater output and map it to sensor 2
M143 H2 S301                                               ; set temperature limit for heater 2 to 301C

M308 S3 P"100.temp1" Y"thermistor" T100000 B4725 C7.06e-8  ; configure sensor 3 as thermistor
M950 H3 C"100.out1" T3                                     ; create nozzle heater output and map it to sensor 3
M143 H3 S301                                               ; set temperature limit for heater 3 to 301C

M308 S4 P"100.temp2" Y"thermistor" T100000 B4725 C7.06e-8  ; configure sensor 2 as thermistor
M950 H4 C"100.out2" T4                                     ; create nozzle heater output and map it to sensor 2
M143 H4 S301                                               ; set temperature limit for heater 2 to 301C

M302 S190 R190                                             ;Cold extrusion settings

M307 H1 R4.637 K0.673:0.218 D2.31 E1.35 S1.00 B0 V24.2     ; PID Tuning T0
M307 H2 R4.174 K0.555:0.276 D1.96 E1.35 S1.00 B0 V24.2     ; PID Tuning T1
M307 H3 R4.301 K0.536:0.105 D1.89 E1.35 S1.00 B0 V24.2     ; PID Tuning T2
M307 H4 R3.780 K0.576:0.060 D1.57 E1.35 S1.00 B0 V24.2     ; PID Tuning T3

; Fans
M950 F0 C"0.out7"                                 ; Hotend Fan Tool 0
M106 P0 S255 L255 H1 T50 C"T0 Hotend"             ; Hotend Fan Tool 0
M950 F1 C"0.out4" Q50                             ; Layer Fan Tool 0
M106 P1 S0 H-1 C"T0"                              ; Layer Fan Tool 0

M950 F2 C"0.out8"                                 ; Hotend Fan Tool 1
M106 P2 S255 L255 H2 T50 C"T1 Hotend"             ; Hotend Fan Tool 1
M950 F3 C"0.out5" Q50                             ; Layer Fan Tool 1
M106 P3 S0 H-1 C"T1"                              ; Layer Fan Tool 1

M950 F4 C"100.out5"                               ; Hotend Fan Tool 2
M106 P4 S255 L255 H3 T50 C"T2 Hotend"             ; Hotend Fan Tool 2
M950 F5 C"100.out3" Q50                           ; Layer Fan Tool 2
M106 P5 S0 H-1 C"T2"                              ; Layer Fan Tool 2

M950 F6 C"100.out6"                               ; Hotend Fan Tool 3
M106 P6 S255 L255 H4 T50 C"T3 Hotend"             ; Hotend Fan Tool 3
M950 F7 C"100.out4" Q50                           ; Layer Fan Tool 3
M106 P7 S0 H-1 C"T3"                              ; Layer Fan Tool 3

;Static layer fan
M950 F8 C"0.out0" Q65535                          ; static layer fan
M106 P8 S0 H-1 I1 L0.2 C"RSCS"

; Tools
M563 P0 D0 H1 F1                                 ; define tool 0
G10 P0 R0 S0                                     ; set initial tool 0 active and standby temperatures to 0C

M563 P1 D1 H2 F3                                 ; define tool 1
G10 P1 R0 S0                                     ; set initial tool 1 active and standby temperatures to 0C

M563 P2 D2 H3 F5                                 ; define tool 2
G10 P2 R0 S0                                     ; set initial tool 2 active and standby temperatures to 0C

M563 P3 D3 H4 F7                                 ; define tool 3
G10 P3 R0 S0                                     ; set initial tool 3 active and standby temperatures to 0C

M563 P4 S"Pen"                                   ; define tool 4 (Pen)

;Dock present switches
M950 J0 C"0.io7.in"                              ; Tool 0 Dock switch
M950 J1 C"0.io8.in"                              ; Tool 1 Dock switch
M950 J2 C"100.io5.in"                            ; Tool 2 Dock switch
M950 J3 C"100.io6.in"                            ; Tool 3 Dock switch

;additional tool related switches
;M950 J4 C"0.io3.in"                             ;potential toolhead tool present switch, not installed atm, but already wired up inside the box

;Filament sensors, not installed atm, but already wired up inside the box
;M591 D0 P3 C"100.io0.in" S0
;M591 D1 P3 C"100.io1.in" S0
;M591 D2 P3 C"100.io2.in" S0
;M591 D3 P3 C"100.io3.in" S0

M950 S0 C"0.out9" Q333                          ;PebbleWiper Servo
M950 P1 C"0.io5.out"                            ;Light switch PWM

;Toolhead Dotstar
M150 X0

;Accelerometer
M955 P0 C"spi.cs2+spi.cs3"

;Inputshaping
M593 P"zvddd" F70 S0.2 

; Miscellaneous
M98 P"/macros/Misc/Lights/On"
M98 P"/sys/config-tools.g"

if !exists(global.hasRunInit)
	global pebbleFeedSpeedDefault = 300
	global pebbleFeedSpeed = global.pebbleFeedSpeedDefault	
	global deactivateToolAfterFilamentChange = false
	global hasRunInit = true

M122 B0 (6HC)

M122 B0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.5beta1 (2022-12-23 18:27:08) running on Duet 3 MB6HC v1.02 or later (standalone mode)
Board ID: 08DJM-9P63L-DJ3T0-6J1F4-3S06S-9A1V9
Used output buffers: 1 of 40 (34 max)
=== RTOS ===
Static ram: 151524
Dynamic ram: 111232 of which 60 recycled
Never used RAM 84544, free system stack 148 words
Tasks: NETWORK(ready,32.3%,211) ETHERNET(notifyWait,0.1%,401) ACCEL(notifyWait,0.0%,220) HEAT(notifyWait,0.0%,321) Move(notifyWait,0.0%,265) CanReceiv(notifyWait,0.0%,773) CanSender(notifyWait,0.0%,335) CanClock(delaying,0.0%,340) TMC(notifyWait,8.1%,56) MAIN(running,59.4%,953) IDLE(ready,0.0%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 01:16:44 ago, cause: power up
Last software reset at 2022-12-30 14:56, reason: HardFault unaligned, none spinning, available RAM 84940, slot 2
Software reset code 0x4072 HFSR 0x40000000 CFSR 0x01000000 ICSR 0x00427803 BFAR 0x00000000 SP 0x2041f890 Task NETW Freestk 318 ok
Stack: 20430645 00a5a501 a5000000 204311b0 20428cc8 004438ff 0043befe 21030000 00000000 40490000 00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000 20428cc8 00444053 0044405e 21030000 00000000 407f4000 7b79718d 3f8dae65 00000000
Error status: 0x00
Step timer max interval 828
MCU temperature: min 20.2, current 42.7, max 44.6
Supply voltage: min 23.7, current 23.8, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.2, current 12.3, max 12.5, under voltage events: 0
Heap OK, handles allocated/used 99/38, heap memory allocated/used/recyclable 2048/2000/1340, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 520, reads 35948, writes 23 timeouts 0
Driver 1: standstill, SG min 0, mspos 520, reads 35948, writes 23 timeouts 0
Driver 2: standstill, SG min 0, mspos 296, reads 35932, writes 39 timeouts 0
Driver 3: standstill, SG min 0, mspos 104, reads 35932, writes 39 timeouts 0
Driver 4: standstill, SG min 0, mspos 520, reads 35938, writes 35 timeouts 0
Driver 5: standstill, SG min 0, mspos 520, reads 35950, writes 23 timeouts 0
Date/time: 2022-12-30 22:10:13
Slowest loop: 43.64ms; fastest: 0.04ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.5ms, write time 19.7ms, max retries 0
=== Move ===
DMs created 125, segments created 19, maxWait 1561619ms, bed compensation in use: none, comp offset 0.000
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 54, completed 54, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 41478, received 92112, lost 0, boc 0
Longest wait 1ms for reply type 6018, peak Tx sync delay 32636, free buffers 50 (min 49), ts 23023/23022/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 73.04ms; fastest: 0.03ms
Responder states: HTTP(4) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 0 0 0
Socket states: 5 3 2 2 2 0 0 0
= WiFi =
Network state is disabled
WiFi module is disabled
Failed messages: pending 2779096485, notready 2779096485, noresp 2779096485
Socket states: 0 0 0 0 0 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

M122 B100 (Mini 5+ as expansion)

M122 B100
Diagnostics for board 100:
RepRapFirmware for Duet 3 Mini 5+ version 3.5beta1 (2022-12-23 18:25:34) running on Duet 3 Mini5plus Ethernet
Last reset 01:17:06 ago, cause: power up
Driver 0: 408.0 steps/mm,, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 27334, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: 408.0 steps/mm,, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 27334, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: 408.0 steps/mm,, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 27335, writes 12, timeouts 0, DMA errors 0, CC errors 0
Driver 3: 408.0 steps/mm,, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 27335, writes 12, timeouts 0, DMA errors 0, CC errors 0
Driver 4: 80.0 steps/mm,, SG min 0, read errors 0, write errors 0, ifcnt 10, reads 27338, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 5: 80.0 steps/mm,, SG min 0, read errors 0, write errors 0, ifcnt 10, reads 27338, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 6: 80.0 steps/mm,, SG min 0, read errors 0, write errors 0, ifcnt 10, reads 27338, writes 10, timeouts 0, DMA errors 0, CC errors 0
VIN: 23.7V, MCU temperature: min 19.5C, current 34.9C, max 34.9C
Peak sync jitter -6/7, peak Rx sync delay 202, resyncs 0/0, no step interrupt scheduled

[dc42]

@Diamondback does the same problem occur if you send M999 to reset the main board explicitly?

[Diamondback]

@dc42 Yes, the same happens when sending M999.
A tiny bit more info, when this occurs, the main board takes a very long time to start up (as if it was waiting for the CAN connection to start up or something), eventually it does boot up but has no actual connection to the expansion board (tools that use heaters there show no temps etc)

[dc42]

@Diamondback thanks. I suspect this is a new issue in RRF 3.5. Let me know if you find it happens using 3.4.5 too.

[Diamondback]

@dc42 Just went back to 3.4.5 so that I can actually print and this issue does NOT happen there, so seems to be new with 3.5b1.

[dc42]

@Diamondback thanks for confirming. Did you run G32 on the main board before the problem occurred? If so then please see https://forum.duet3d.com/post/303514.

[Diamondback]

@dc42 No, this happened after a fresh boot.

[dc42]

@Diamondback thanks, I'll try to reproduce it.

[dc42]

@Diamondback I have reproduced it:

• Doing a software reset of the main board causes the expansion board to lose CAN connection;
• Pressing the reset button on the expansion board restores the connection;
• Pressing the reset button on the main board causes only a temporary loss of CAN connection.

[dc42]

@Diamondback I have fixed this, along with the problem of not being able to update a main board used as an expansion board over CAN. I have put a new Mini5+ binary at https://www.dropbox.com/sh/28zqy8t165nwa2f/AACZvmVSGhWMSpjKX43MsSWSa?dl=0.

[Diamondback]

@dc42 awesome, will test soon! Thanks a lot!