CAN: Warning, Error and timeout. During start of print only

dintid

Story
Been printing a lot the past week without issues. I've made no changes at all on the printer, and no changes in slizer prior to the issue:
In other words: It used to work perfectly, so BLTouch is not a new addition.

Symptoms:
What it does now it probe middle of bed, idle out on front left, then go and probe rear center, then go idle out front right... does it 3 times.. and then it goes on to print just fine.
It's consistent in this and I cannot get it to fail when I do a selftest on BLTouch or just deploy and retract it etc..
The CAN status light is blinking as expected and it's only complaining during the startup sequence...
I have had it turned off, and also waited till next day with power off without any change.

Machine type
Using a cube style CoreXY machine with 3 seperate Z.

Hadware and firmware
I'm using Duet 3 in SBC (pi4 2gb) mode with a Toolboard LC (v1.0), both at FW 3.3
Using Toolboard Distribution board.
Wires used from distribution board to Toolboard are twisted silicone wires.
I'm using original BLTouch v3.1
Original Hemera extruder and some Sonun fans.

Startup sequence
I have a pretty straightforward simple startup sequence without conditional statements, but simply just calling old fashioned G28 from slizer to do the homing which do the probing in the middle of bed using G30, where it raises Z bed to nozzle and finally G32 where it go to front left corner and probe, then rear center and finally front right.
I do this 3 point probing 3 times total.
Files attached homeall.g config.g bed.g

M122
RepRapFirmware for Duet 3 MB6HC version 3.3 (2021-06-15 21:45:47) running on Duet 3 MB6HC v1.01 or later (SBC mode)

M122 B121
Diagnostics for board 121:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Notes: hardware version v1.0

When I updated to FW 3.3 I had to upgrade Toolboard LC (v1.0) firmware as well. I cant' find the link again, but I didn't get the very latest version as it listed that some things weren't tested yet.

homeall.g config.g bed.g

dc42

@dintid please can you upgrade to 3.4.0rc1 by switching t the unstable feed on the package server, and test again.

dintid

@dc42 I can't seem to find the page where it's descriped how to change to unstable repo. Link me?
I have been using FW 3.3 daily for a week without this issue, so it's a somewhat odd. Af first I thought it was a loose connection, but it not anymore.

Phaedrux

https://github.com/Duet3D/DuetSoftwareFramework/wiki/SBC-Setup-Guide#unstable-package-feed

dintid

@dc42 updated without issues, but it seems to have made matters worse.

Noticed hotend sensor (PT1000) switch between normal reading and 2000c. Heatsink fan does turn on and off whenever sensor burbs, so some information is still sent.

The problem with homing (G32) is not 100% the same each time now as before RC fw.

I noticed the CAN led blinks in sync, but when it goes to probe front left and front right it starts blinking rapidly.
Each time it goes to center rear while it blinks rapidly, it probes and the blinking slows down untill it gets to front right, at which point it speeds up and fails.

I've had it even fail normal G30 now. It never did that before.

In a startup sequence I just did the sensor started blinking rapidly as soon as Z (bed) began lift. Same time sensor went to 2000c. Once it was supposed to probe it stopped blinking entirely and failed all probes.
Had to turn the machine off and on.

Status:
M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.0 or earlier firmware version 3.4.0rc1 (2022-02-08 09:04:48)
Bootloader ID: not available
All averaging filters OK
Never used RAM 3576, free system stack 88 words
Tasks: Move(notifyWait,0.0%,153) HEAT(notifyWait,0.2%,115) CanAsync(notifyWait,0.0%,65) CanRecv(notifyWait,0.0%,76) CanClock(notifyWait,0.0%,65) TMC(notifyWait,3.0%,57) MAIN(running,92.0%,341) IDLE(ready,0.0%,40) AIN(delaying,4.8%,142), total 100.0%
Last reset 00:01:38 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 390.0 steps/mm,standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 49128, writes 11, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 4/8, peak Rx sync delay 547, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 24.2, current 24.2, max 24.3
MCU temperature: min 37.3C, current 37.5C, max 37.8C
Last sensors broadcast 0x00000002 found 1 63 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 1715, send timeouts 0, received 815, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer: none
I2C bus errors 4619, naks 4619, other errors 4619

dintid

@dc42 After power cycle it seems to not find anything (heater, fans or bltouch) on Toolboard.
Can led blinks in sync with mainboard

dc42

@dintid said in CAN: Warning, Error and timeout. During start of print only:

I noticed the CAN led blinks in sync, but when it goes to probe front left and front right it starts blinking rapidly.
Each time it goes to center rear while it blinks rapidly, it probes and the blinking slows down untill it gets to front right, at which point it speeds up and fails.

That indicates an intermittent break in the CAN cable to the tool board.

dintid

@dc42 said in CAN: Warning, Error and timeout. During start of print only:

intermittent

Before upgrading to RC it happened 100% always at the same point for front left and right. After firmware upgrade it got sporadic.

Now Toolboard just flashes very quick and no response.
I've tested wires from Distribution board to Toolboard for continuity, and they test fine. Also tested the RJ11 cable from Duet3 to Distribution board (using the supplied one).
Not using any plugins.
Downgraded firmware again but no change in the above.

dc42

@dintid I'm still sure that you have a bad connection in the CAN cable, or possibly in the soldering of the CAN connector to the tool board, or just possibly in the power cable to the tool board. It's coincidence that it got worse around the time you upgraded the firmware. The connection works or not depending on which way the cable bends. I had exactly this problem in one of my tool board cables too.

dintid

@dc42 With great experience comes great helping power.. or something like that
Turned out the wires inside crimps on one end just hung on by a thread (literally). Since I had to test for continuity from the point where the wires were inserted into connector I didn't catch this.

I downgraded firmware and did all the commands as listed on github (as @Phaedrux linked). I had to manually flash the toolboard, but they are both at fw 3.3 now.. my pi-screen doesn't boot up with preloaded webpage for my printer now though. (I used precompiled image you made back when I first set it up).
Any easy fix for this?
Also I noticed, prior to all this, how the webbrowser on pi-screen always complained that it wasn't shut down properly, so had to reload after each power-on of the machine.
Is there a shutdown procedure, or other way to overcome this?

Phaedrux

When was the last time you created a fresh SD card with a recent DuetPi image?

It may be a good idea to backup your config files and create a new one, or better yet, use a new SD card to keep your old as a functional backup.

https://docs.duet3d.com/en/User_manual/Machine_configuration/SBC_setup

dintid

@phaedrux the missing Duet webpage is not due to SD corruption (or I'll be extremely surprised).It happened as I went to RC and back again. I didn't check pi-screen so don't know if it happens during move to RC or back again.
I'm using high end SD card, and while I set it in june 2020 (I wrote this while at it https://betrue3d.dk/rpi-and-duet-3-why-and-how/ ) I havn't ever used this machine as I've moved around some.
It's only printed like max 1kg of filament or so.
I'm doing backups before making changes.

dintid

@phaedrux as a sidenote.. are the docs subdomain section meant to take over for the dozuki?

Phaedrux

@dintid said in CAN: Warning, Error and timeout. During start of print only:

@phaedrux as a sidenote.. are the docs subdomain section meant to take over for the dozuki?

Yes, the new documentation site has recently become available at https://docs.duet3d.com/

Dozuki will eventually go away.

The reason I asked about the last time you did DuetPi image is that the image has changed over time.