[3.4.5] CAN connection drops mid-print (OoM error)

Diamondback

It appears that "free system stack" also decreases somewhat over time, much much slower than the never used RAM though. It is now stuck at 120 words since some time.

Btw, here's what I have connected to the slave Mini 5: (I am printing with T1 here, which is fully controlled from the 6HC master)

4x Extruder, one of them being used in the actual print
Bed heater/temp, used in the print
Heater/temp T2
Heater/temp T3
Hotend- & partcooling fans T2
Hotend- & partcooling fans T3
"Tool-at-dock"-switch T2
"Tool-at-dock"-switch T3

Diamondback

Here's a few example M122 outputs from before and during the print:

Fresh after boot:

1.1.2023, 16:11:03	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.5 (2022-11-30 19:41:16) running on Duet 3 Mini5plus Ethernet (expansion mode)
Board ID: H1590-0296U-D65J0-40KMQ-JP03Z-RGPPB
Used output buffers: 1 of 40 (18 max)
=== RTOS ===
Static ram: 103652
Dynamic ram: 104764 of which 0 recycled
Never used RAM 33296, free system stack 200 words
Tasks: NETWORK(ready,27.1%,254) ETHERNET(notifyWait,0.2%,560) HEAT(notifyWait,0.0%,358) Move(notifyWait,0.0%,363) CanReceiv(notifyWait,0.0%,654) CanSender(notifyWait,0.0%,336) CanClock(invalid,0.0%,351) TMC(notifyWait,1.2%,116) MAIN(running,70.1%,719) IDLE(ready,0.6%,30) AIN(delaying,0.8%,263), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:00:20 ago, cause: power up
Last software reset at 2023-01-01 16:10, reason: User, GCodes spinning, available RAM 34580, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 21306, completed 21305, timed out 0, errs 0
Step timer max interval 752
MCU temperature: min 32.7, current 35.1, max 35.4
Supply voltage: min 23.7, current 23.7, max 23.8, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 1838, writes 11, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 1837, writes 12, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 1836, writes 12, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 1837, writes 11, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 1840, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 1840, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 1839, writes 9, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-01-01 16:11:03
Cache data hit count 35445150
Slowest loop: 2.14ms; fastest: 0.13ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 1.9ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 420, received 212, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 222, free buffers 18 (min 18), ts 2/1/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 5.85ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0

Some time into the print:

1.1.2023, 16:30:36	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.5 (2022-11-30 19:41:16) running on Duet 3 Mini5plus Ethernet (expansion mode)
Board ID: H1590-0296U-D65J0-40KMQ-JP03Z-RGPPB
Used output buffers: 1 of 40 (31 max)
=== RTOS ===
Static ram: 103652
Dynamic ram: 104764 of which 0 recycled
Never used RAM 31256, free system stack 140 words
Tasks: NETWORK(ready,28.0%,254) ETHERNET(notifyWait,0.3%,560) HEAT(notifyWait,0.0%,342) Move(notifyWait,0.0%,338) CanReceiv(notifyWait,0.1%,650) CanSender(notifyWait,0.0%,336) CanClock(invalid,0.0%,351) TMC(notifyWait,1.2%,72) MAIN(running,69.5%,637) IDLE(ready,0.0%,30) AIN(delaying,0.8%,263), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:19:53 ago, cause: power up
Last software reset at 2023-01-01 16:10, reason: User, GCodes spinning, available RAM 34580, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 1194262, completed 1194261, timed out 0, errs 0
Step timer max interval 736
MCU temperature: min 37.1, current 37.1, max 37.1
Supply voltage: min 23.5, current 23.6, max 23.6, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 102, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: ok, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 103, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 103, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 103, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 103, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 103, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 103, writes 0, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-01-01 16:30:43
Cache data hit count 2028212819
Slowest loop: 1.15ms; fastest: 0.13ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 85, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 5124, completed 5123, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 1], CDDA state 3
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 20, received 39, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 18 (min 18), ts 0/0/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 5.47ms; fastest: 0.04ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0

A 24 byte drop:

1.1.2023, 16:43:19	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.5 (2022-11-30 19:41:16) running on Duet 3 Mini5plus Ethernet (expansion mode)
Board ID: H1590-0296U-D65J0-40KMQ-JP03Z-RGPPB
Used output buffers: 1 of 40 (32 max)
=== RTOS ===
Static ram: 103652
Dynamic ram: 104764 of which 0 recycled
Never used RAM 3008, free system stack 120 words
Tasks: NETWORK(ready,27.5%,254) ETHERNET(notifyWait,0.3%,560) HEAT(notifyWait,0.1%,342) Move(notifyWait,0.0%,338) CanReceiv(notifyWait,0.0%,638) CanSender(notifyWait,0.0%,336) CanClock(invalid,0.0%,351) TMC(notifyWait,1.2%,72) MAIN(running,70.0%,637) IDLE(ready,0.0%,30) AIN(delaying,0.8%,263), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:32:36 ago, cause: power up
Last software reset at 2023-01-01 16:10, reason: User, GCodes spinning, available RAM 34580, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 1957490, completed 1957490, timed out 0, errs 0
Step timer max interval 647
MCU temperature: min 37.8, current 38.1, max 38.2
Supply voltage: min 23.6, current 23.7, max 23.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 107, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: ok, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 107, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 107, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 107, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 108, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 107, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 107, writes 0, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-01-01 16:43:26
Cache data hit count 3316121938
Slowest loop: 1.22ms; fastest: 0.13ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 1262, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 30310, completed 30309, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state 3
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 25, received 19, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 18 (min 18), ts 0/0/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 4.95ms; fastest: 0.04ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0

To:

1.1.2023, 16:43:21	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.5 (2022-11-30 19:41:16) running on Duet 3 Mini5plus Ethernet (expansion mode)
Board ID: H1590-0296U-D65J0-40KMQ-JP03Z-RGPPB
Used output buffers: 1 of 40 (32 max)
=== RTOS ===
Static ram: 103652
Dynamic ram: 104764 of which 0 recycled
Never used RAM 2984, free system stack 120 words
Tasks: NETWORK(ready,27.7%,254) ETHERNET(notifyWait,0.3%,560) HEAT(notifyWait,0.1%,342) Move(notifyWait,0.0%,338) CanReceiv(notifyWait,0.1%,638) CanSender(notifyWait,0.0%,336) CanClock(invalid,0.0%,351) TMC(notifyWait,1.2%,72) MAIN(running,69.8%,637) IDLE(ready,0.0%,30) AIN(delaying,0.8%,263), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:32:38 ago, cause: power up
Last software reset at 2023-01-01 16:10, reason: User, GCodes spinning, available RAM 34580, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 1958782, completed 1958781, timed out 0, errs 0
Step timer max interval 679
MCU temperature: min 37.8, current 38.1, max 38.2
Supply voltage: min 23.6, current 23.6, max 23.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 117, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: ok, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 118, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 118, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 117, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 117, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 118, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 117, writes 0, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-01-01 16:43:28
Cache data hit count 3318307830
Slowest loop: 1.15ms; fastest: 0.13ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 1263, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 30328, completed 30326, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state 3
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 25, received 30, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 18 (min 18), ts 0/0/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 4.85ms; fastest: 0.04ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0

Last one before it died:

1.1.2023, 16:44:24	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.5 (2022-11-30 19:41:16) running on Duet 3 Mini5plus Ethernet (expansion mode)
Board ID: H1590-0296U-D65J0-40KMQ-JP03Z-RGPPB
Used output buffers: 1 of 40 (32 max)
=== RTOS ===
Static ram: 103652
Dynamic ram: 104764 of which 0 recycled
Never used RAM 344, free system stack 120 words
Tasks: NETWORK(ready,27.9%,254) ETHERNET(notifyWait,0.3%,560) HEAT(notifyWait,0.1%,342) Move(notifyWait,0.1%,338) CanReceiv(notifyWait,0.2%,638) CanSender(notifyWait,0.0%,336) CanClock(invalid,0.0%,351) TMC(notifyWait,1.2%,72) MAIN(running,69.4%,637) IDLE(ready,0.0%,30) AIN(delaying,0.8%,263), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:33:41 ago, cause: power up
Last software reset at 2023-01-01 16:10, reason: User, GCodes spinning, available RAM 34580, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 2021886, completed 2021884, timed out 0, errs 0
Step timer max interval 655
MCU temperature: min 38.1, current 38.1, max 38.2
Supply voltage: min 23.6, current 23.6, max 23.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 109, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 1: ok, SG min 0, read errors 0, write errors 0, ifcnt 14, reads 108, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 108, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 109, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 109, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 108, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 108, writes 0, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2023-01-01 16:44:31
Cache data hit count 3424759447
Slowest loop: 1.17ms; fastest: 0.13ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 1373, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 32545, completed 32539, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state 3
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 25, received 81, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 18 (min 18), ts 0/0/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 4.96ms; fastest: 0.04ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 0 0 0
Socket states: 5 5 2 2 2 0 0 0

Diamondback

As mentioned, I did complete a 2.5h print before with this setup, so the issue does not appear to be purely related to time.

The biggest difference I see between the working and failed print is that the working one has much longer extrusion segments and the failing one probably has a lot more tiny segments and maybe also a lot more retracts and possibly more PA action as well.

dc42

@Diamondback thanks. Are any of the endstops or the Z probe connected to the Duet 3 Mini?

Diamondback

@dc42 no, I made sure that all axis endstops and the Z probe are connected to the same board as the relevant steppers (ie all on the 6HC master)

dc42

@Diamondback thanks again. I am looking for data structures that consume 24 bytes of memory, and endstops and Z probes were one candidate.

Is it possible that the reduction in memory is correlated with operation of the tool-at-dock switches? Bear in mind that there may be contact bounce when they operate.

Diamondback

@dc42 no, these switches are not operated during the print at all (and I don't have any evidence that some sort of vibration would constantly trigger them by accident).
Maybe it's 3*8 bytes instead or something like this...

dc42

@Diamondback please install the firmware at https://www.dropbox.com/sh/wmrfidzkej24mcm/AAB7AFGrfMoYqssjbVKM5rwxa?dl=0 on the Mini5+ and monitor part of a print with M122 as before. I have added new fields heap top and heap limit to the report. When it has settled down, I expect either heap top to increase by 24 byes at a time, or heap limit to decrease by 24 bytes at a time. I need to know which.

dc42

@Diamondback ~~further to my previous message, I have found and fixed a possible cause of the memory leak. Please try the firmware at https://www.dropbox.com/sh/ihw972tlfq3gmj3/AABxTz_F0RYRC3i124PL4Wvma?dl=0.~~ Cancel that, I need to make a change to the fix.

dc42

@Diamondback second attempt: new firmware with possible memory leak fix at https://www.dropbox.com/sh/ihw972tlfq3gmj3/AABxTz_F0RYRC3i124PL4Wvma?dl=0.

Diamondback

@dc42 Which particular 6HC firmware should I use in combination with this? Is your other custom build from the reset/update via CAN issue ok?

Diamondback

Trying the fixed one first before the one with more debug output. Is the never used RAM supposed to be perfectly stable? It still does go down by 24bytes at a time, however a LOT slower than before.
Only lost 72 bytes in the last 6 minutes now and only 480 since I started the print 11 minutes ago.

Diamondback

Looks like it stopped decreasing now at 28076, the print finished without issues. Now trying to print a second time without any reset.

dc42

@Diamondback yes when you first print, expect the free memory to go down and then stabilise. It may go down in two sizes of step, one will be 24 (max. 120 steps) and the other will be greater (max. 40 steps). If you do a second print, it should not go down any more.

Diamondback

@dc42 After a few consecutive prints without resets, it still sits at 28076. So I guess this is fixed! Awesome work, thanks a lot!