Toolboard v1.1 loses connection with mainboard

[kingofthegeeks]

This started happing about a week ago, the first time it happened 48 hours into a 115 hour print, these long prints are regular for me and this is literally the first time in a year I've had a print fail in the middle .

I thought it was erroneous and started another print, this one failed the same way at the 15 hr. mark. I started troubleshooting, changing heaters and a few different thermistors, each time the failure would happen quicker till now where it will usually happen within three probes of a mesh bed levelling, of course because I typed that it took almost 20 probes before it failed the most recent time.

12/30/2021, 4:22:42 PM: G28
12/30/2021, 4:22:51 PM: G1 Z10
12/30/2021, 4:22:58 PM: G1 X200 Y200
12/30/2021, 4:23:18 PM: G30
probes 20 points before connection drops
12/30/2021, 4:26:49 PM: G29: Error: Board 20 does not have input handle 3000
Error: Failed to enable probe

M122's shortly after failure

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.0beta7+2 (2021-12-30 17:36:30) running on Duet 3 MB6HC v1.01 or later (standalone mode)
Board ID: 08DJM-956L2-G43S8-6JTDG-3S86Q-KB3QF
Used output buffers: 1 of 40 (15 max)
=== RTOS ===
Static ram: 150728
Dynamic ram: 95204 of which 0 recycled
Never used RAM 104668, free system stack 150 words
Tasks: NETWORK(ready,26.8%,253) ETHERNET(notifyWait,0.1%,169) HEAT(notifyWait,0.0%,324) Move(notifyWait,0.0%,248) CanReceiv(notifyWait,0.0%,797) CanSender(notifyWait,0.0%,356) CanClock(delaying,0.0%,339) TMC(notifyWait,7.7%,58) MAIN(running,65.3%,1117) IDLE(ready,0.0%,30), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:05:50 ago, cause: software
Last software reset at 2021-12-30 16:21, reason: User, GCodes spinning, available RAM 104644, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0044a000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Step timer max interval 171
MCU temperature: min 50.2, current 51.2, max 51.3
Supply voltage: min 26.5, current 26.5, max 26.6, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.3, under voltage events: 0
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: pos 9700, standstill, SG min 0, mspos 8, reads 56923, writes 19 timeouts 0
Driver 1: pos 19900, standstill, SG min 0, mspos 504, reads 56923, writes 19 timeouts 0
Driver 2: pos 6160, standstill, SG min 0, mspos 840, reads 56923, writes 19 timeouts 0
Driver 3: pos 96950, standstill, SG min 0, mspos 424, reads 56924, writes 19 timeouts 0
Driver 4: pos 0, standstill, SG min 0, mspos 744, reads 56924, writes 19 timeouts 0
Driver 5: pos 0, standstill, SG min 0, mspos 8, reads 56932, writes 11 timeouts 0
Date/time: 2021-12-30 16:27:50
Slowest loop: 7.23ms; fastest: 0.05ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.6ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 3, maxWait 29750ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 127, completed 127, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 2], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 3316, received 13452, lost 0, boc 0
Longest wait 3ms for reply type 6053, peak Tx sync delay 51, free buffers 50 (min 49), ts 1751/1750/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 6.00ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 5 5 2 2 2 0 0 0

Diagnostics for board 20:
Duet TOOL1LC rev 1.1 or later firmware version 3.4.0beta7+2 (2021-12-30 22:07:52)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 3072, free system stack 2788 words
Tasks: Move(notifyWait,0.0%,153) HEAT(notifyWait,0.1%,115) CanAsync(notifyWait,0.0%,65) CanRecv(notifyWait,0.0%,76) CanClock(notifyWait,0.0%,65) ACCEL(notifyWait,0.0%,61) TMC(delaying,3.0%,57) MAIN(running,92.0%,441) IDLE(ready,0.0%,26) AIN(delaying,4.9%,142), total 100.0%
Last reset 00:01:07 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm,standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 33876, writes 9, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 6/10, peak Rx sync delay 198, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 26.2, current 26.6, max 26.6
MCU temperature: min 34.9C, current 35.0C, max 35.3C
Last sensors broadcast 0x00000000 found 0 51 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 565, send timeouts 0, received 891, lost 0, free buffers 37, min 37, error reg 80000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 3, other errors 0

I've already ordered a toolboard so I can remove and replace with everything else in place, I guess I'm looking for confirmation tests while I wait as it seems terribly odd to me.

KOTG

[dc42]

@kingofthegeeks the M122 report for the tool board says it was power cycled just over 1 minute before you took the report. Is that what you did?

When communication with the tool board had been lost, what was the red Status LED on the tool board doing?

[kingofthegeeks]

@dc42 No, it appears I missed that reboot, which now that I pay closer attention is exactly appears to be happening when it loses coms. I just went through the process twice and if it were just for the lights on the tool board wouldn't even notice but the fan comes on for a fraction of a second and I get the error I received previously.

The tool board appears to immediately sync back up with the main board although DWC shows it offline, and any process is lost.

Its an IDEX machine and I just ran through the process on tool 2 with no problems just for my brain to confirm the main board is good. Its another tool board but with a v1.0

Thanks for paying attention to my little issue, I should have better investigated, but your gear has worked so well I rarely have to bother troubleshooting and it made me lazy.

Mike

[dc42]

@kingofthegeeks so was the problem an intermittent power connection to the tool board?

[kingofthegeeks]

@dc42 I don't believe so.

I did however take this opportunity waiting for a new toolboard to rewire everything to be sure, in that process now.

I'll expect to know more soon but I feel like an idiot not noticing the power loss and I'll wait to have something compelling before I run my mouth again.

Mike