Firmware update bricked printer - SPI connection has been reset

p8blr

This a continuation of this thread, as it no longer pertains to the brakes not engaging, rather an entire system failure.

Configuration:
(1) 6XD
(5) 3HC
(1) 1XD
Raspberry Pi 5 (8GB) with a NVME SSD hat
3.5.2 firmware

Backstory:
This is a large 3d printer with many duet boards running in sbc mode with a raspberry pi 5 and nvme ssd hat. It has been running in it's current configuration for over a year now with no major hiccups. One issue I've been experiencing was the brakes not engaging/disengaging properly, so I attempted to use the .bin file from the linked thread provided by @timschneider. I installed the firmware over the network via DWC by uploading it to the firmware directory. Immediately after that I started getting a flurry of SPI connection reset issues.

Things I've tried that did not work:

I saw one post that recommended running sudo rpi-update. This did not resolve the issue, however, now I am at the latest pi eeprom firmware.
I tried running M997 F"stable", then M997 S2 but it didn't appear to do anything.
Installing a newly flashed NVME with the non lite 64 bit image available here
Reflashing the NVME and running sudo apt update, sudo apt upgrade, sudo apt-full upgrade
Downloading the latest bootloaders from here and installing them with this method.
Using an alligator clip connected to the USB shield of the RPI to earth ground
Reflashing a new NVME with the latest RPI image, M997 S2 F"3.5.2"
Setting up a new microSD card with my files and trying to use the Duet without SBC mode, however, DWC crashes, freezes, and refreshes so much that it's useless and I didn't continue.
creating a simplified version of config.g and driveSetup.g that do not reference anything outside of the 6XD, and disconnecting the CAN cable. After uploading and running the config file I get "SPI connection has been reset" but no "driver does not exist" errors. Interestingly, now DWC doesn't open on the RPI, but the web interface does work through a browser on a networked PC.
Disconnecting the NVME hat and flashing a fresh microSD card instead, with no CAN connected, and the simplified file structure, still has SPI errors. I think this narrows it down to something being wrong with the 6XD. Which goes back to me using this file to upload via DWC and originally cause this error.
This is an edit because nobody has responded yet, but I took a spare 6XD that I had, re-imaged the NVME, connected everything, ran sudo apt-get update, sudo apt-get upgrade (this updated firmware on the 6XD to 3.5.2) and I still get the same SPI errors. Perhaps I will now try downgrading firmware...

Interesting notes:

M115 B0, B1, B2, B3, B4, B5, 121, 122 all show that the boards are connected. Also the red CAN leds blink in sync.
I can update firmware on each board with no problem.
Deleting my config.g or using the stock config.g that does not reference any CAN boards does not result in SPI issues.
If I run M98 P"config.g" after the printer has started up, nothing happens

I don't believe this is a shielding issue since the printer has been running fine for so long up until the update. And as I mentioned, adding a ground wire to the RPI didn't help. I do have an oscilloscope that I can use to check whatever you would like me to. I don't think I can restore the RPI to it's old bootloader because I don't know what it used to be running. I don't have a copy of the old RPI image because I got it from a forum post when bookworm was just released. I have many copies of my config.g and related files, however, there are no major changes as the hardware has remained consistent.

Please help as I am at a total loss at this point. Thank you.

config.g
driveSetup.g
startup.g
variables.g

Edit: more updates
Downgraded to 3.4.6, still didn't work, googled and found spi issues can be power related so I tried this:

sudo rpi-eeprom-config --edit
PSU_MAX_CURRENT=5000

and that fixed it. My config file works, although for some reason random variables weren't defined, which makes no sense. So I tried M997 F"unstable", then M997 S2 V"3.5.2-rc.1" and it didn't work. So then I went to terminal and tried sudo apt update, sudo apt upgrade, and I it failed to connect there during the firmware upgrade. So I reboot, and now I notice that on DWC, which decided to open this time, it now says "Failed to connect to localhost" and I know for sure I read somewhere that someone else got their duet messed up with a "fancy" name, but I can't find that post...

Edit: Well I figured it out! Sort of. I don't know why but commenting out these lines:

; Brakes
;M569.7 P0.2 C"out3" S1000               ; driver 2 on board 6XD uses port out3 to control the brake - was a 10ms delay but increased to a second to help with position recovery
;M569.7 P0.3 C"out4" S1000               ; driver 3 on board 6XD uses port out4 to control the brake
;M569.7 P0.4 C"out5" S1000               ; driver 4 on board 6XD uses port out5 to control the brake
;M569.7 P0.5 C"out6" S1000               ; driver 5 on board 6XD uses port out6 to control the brake

fixed the SPI issues, and now I'm running 3.5.2.

I attempted M997 F"unstable" and got "failed to flash flash firmware. Please install it manually" so that's unsettling as I may have broken it again. M997 in SBC mode REALLY need some reliability improvement.

Edit: I had to use Bossa to restore the firmware on the 6XD, and with the brake code still commented out it's working just fine, minus the brakes. I still don't understand how M997 is supposed to work
M997 F"unstable" - "please wait while updates are installed" and then the duet reboots. Nothing happened
M997 S2 - same thing
M997 F"unstable-3.5.3-rc.1" - Error...could not find...
If I try to upload firmware or a DWC zip via DWC it warns me not to b/c I'm in DWC mode, and considering what happened last time, and the fact that m997 exists, I refuse to do that.

I'm quitting for the weekend, see you guys on Tuesday.

timschneider

@p8blr hi sorry to here that it is not working!
I've looked at the config and maybe you can increase the wait delay for the startup of the can connected boards.

; Wait a moment for the CAN expansion boards to start
G4 S2

just increase it for test purpose e.g. to 10 sec. as you can query the boards after startup without an issue but the drive setup fails. I use 5 seconds for one connected expansion board.

Can you also post the content of the daemon.g ~~, as it appears, that there is a problem in line 8. bedStripTempMax, isn't it supposed to be global.bedStripTempMax?~~ Edit: only different output e.g. set global.unknownVar or echo {global.unknownVar}

Do you run your printer mainboard and expansion boards at 24V or more than that? ~~If so, there is a feature in the FW that reduces the voltage via PWM for the breaks and that may create EMV problems.~~ EDIT: it is only activated when M569.7 V is provided, which is not the case.

Maybe you can show the output of M122

btw.

thanks for sharing!

sudo rpi-eeprom-config --edit
PSU_MAX_CURRENT=5000

EDIT
from this screenshot https://forum.duet3d.com/assets/uploads/files/1725041834892-m115.jpg
it is not clear which version runs on the 6XD can you please show the whole line of M115 B0

Phaedrux

Can you use the USB bossa firmware flashing instructions to reflash the 6Xd with the stock 3.5.2 firmware? Then attempt to setup the board standalone by setting up an SD card with the 3.5.2 DWC files without any other canbus boards connected. If that works, go from there and add the expansion boards.

https://docs.duet3d.com/en/User_manual/RepRapFirmware/Updating_firmware#fallback-procedure-2

@p8blr said in Firmware update bricked printer - SPI connection has been reset:

This is an edit because nobody has responded yet, but I took a spare 6XD that I had, re-imaged the NVME, connected everything, ran sudo apt-get update, sudo apt-get upgrade (this updated firmware on the 6XD to 3.5.2) and I still get the same SPI errors.

This is a little strange and makes it sound like it's actually an issue with the ribbon cable or Pi itself.

p8blr

@timschneider Thanks for your reply!

I'll change it to G4 S10

daemon.g

Everything at 24V

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6XD version 3.5.2 (2024-06-11 17:13:17) running on Duet 3 MB6XD v1.0 (SBC mode)
Board ID: 08DLM-956DA-M2NS4-6JTD6-3S86M-1B3GT
Used output buffers: 1 of 40 (24 max)
=== RTOS ===
Static ram: 153768
Dynamic ram: 95836 of which 0 recycled
Never used RAM 93460, free system stack 163 words
Tasks: SBC(2,ready,1.0%,805) HEAT(3,nWait 6,0.0%,367) Move(4,nWait 6,0.0%,211) CanReceiv(6,nWait 1,0.1%,794) CanSender(5,nWait 7,0.0%,329) CanClock(7,delaying,0.0%,351) MAIN(2,running,98.7%,444) IDLE(0,ready,0.1%,29), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:22:18 ago, cause: power up
Last software reset at 2024-08-30 16:53, reason: User, Gcodes spinning, available RAM 100512, slot 2
Software reset code 0x6003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
MCU temperature: min 20.7, current 35.2, max 35.5
Supply voltage: min 24.0, current 24.3, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/48, heap memory allocated/used/recyclable 2048/904/84, gc cycles 0
Events: 3 queued, 3 completed
Driver 0: ok
Driver 1: ok
Driver 2: ok
Driver 3: ok
Driver 4: ok
Driver 5: ok
Date/time: 2024-08-30 18:01:44
Slowest loop: 2.84ms; fastest: 0.06ms
=== Storage ===
Free file entries: 20
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 3, maxWait 25922ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 3, completed 3, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters 1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x20000003
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 12107, received 127720, lost 0, errs 1, boc 0
Longest wait 3ms for reply type 6060, peak Tx sync delay 380, free buffers 50 (min 48), ts 6691/6690/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 52433/52433
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x256f8
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server version 3.5.2 (2024-06-12 07:12:47, 64-bit)
HTTP+Executed:
> Executing M122
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 0
Full transfers per second: 39.69, max time between full transfers: 40.7ms, max pin wait times: 32.3ms/2.7ms
Codes per second: 0.19
Maximum length of RX/TX data transfers: 6102/988

M115 B0
FIRMWARE_NAME: RepRapFirmware for Duet 3 MB6XD FIRMWARE_VERSION: 3.5.2 ELECTRONICS: Duet 3 MB6XD v1.0 FIRMWARE_DATE: 2024-06-11 17:13:17

Thanks!

p8blr

@Phaedrux

So know it may not have been super clear with my scatter-brained posts, but that was actually something I tried. I got the same SPI errors in standalone mode. That last section of brake code that I posted is what I determined set it off. I discovered it by clearing config.g, and slowing adding in my code one section in a time.

p8blr

@dc42 What am I doing wrong?

I also tried the old method:

sudo rm -f /etc/apt/sources.list.d/duet3d-unstable.list
sudo bash -c "echo 'deb https://pkg.duet3d.com/ stable armv7' > /etc/apt/sources.list.d/duet3d.list"
sudo apt update
sudo apt dist-upgrade

But that also didn't upgrade the firmware to 3.5.3-rc.1

p8blr

@chrishamm how do I update to 3.5.3-rc.1 in sbc mode? Can you please send me the exact commands? I’ve tried all possible combinations and it doesn’t work. I updated in standalone mode, but when I connected my pi and tried to audio apt update/upgrade it downgraded my 6xd without asking me. I also have no way of manually updating DWC.

jay_s_uk

@p8blr it looks like 3.5.3-rc1 isn't currently on the DSF package feed so you can't update to it in SBC mode

p8blr

@jay_s_uk Interesting. How did you check that so I know for future reference? Is there no manual way to update firmware in SBC mode?

jay_s_uk

@p8blr was reported in the other thread. And you shouldn't manually update as there may be breaking changes between the firmware and DSF

timschneider

@p8blr was the extended delay able to get rid of the unknown driver messages?

the M122 looks normal to me.

p8blr

@timschneider The delay didn't make a difference, however, changing the brake code to S10 from S1000 seems to have fixed that particular issue.

M569.7 P0.2 C"out3" S10 ; driver 2 on board 6XD uses port out3 to control the brake
M569.7 P0.3 C"out4" S10 ; driver 3 on board 6XD uses port out4 to control the brake
M569.7 P0.4 C"out5" S10 ; driver 4 on board 6XD uses port out5 to control the brake
M569.7 P0.5 C"out6" S10 ; driver 5 on board 6XD uses port out6 to control the brake

Note that I was playing with increasing that value because my servos were getting position recovery errors.

chrishamm

@p8blr It's now available, looks like the GitHub Actions workflow for prereleases wasn't triggered correctly. Sorry for the inconvenience.

p8blr

@chrishamm Didn't work.

Process:
Removed SD card from duet, connected RPI ribbon cable, powered on, M997 S2 F"unstable", sudo apt update, sudo apt full-upgrade

chrishamm

@p8blr /dev/gpiochip4 is correct for the Pi 5. Is it still the same board or did you replace it?

p8blr

@chrishamm Exact same board. I'll reimage the pi and try again.

chrishamm

@p8blr If it's actually caused by a system update, don't use apt to install the latest system package versions, instead use M997 S2 only to update DSF. I'll investigate.

p8blr

@chrishamm new image with no updates works. M997 S2 F"unstable", M997 S2

oozeBot

@chrishamm said in Firmware update bricked printer - SPI connection has been reset:

@p8blr If it's actually caused by a system update, don't use apt to install the latest system package versions, instead use M997 S2 only to update DSF. I'll investigate.

@chrishamm - any updates one what has caused this behavior to start happening? We hadn't noticed it until a recent apt upgrade (not Duet related). Are there workarounds to get this back on the command line? Thanks

chrishamm

@oozeBot We have some more background information about the cause of this problem; it appears to affect the 6HC/6XD only. @dc42 is going to improve the upgrade mechanism in 3.6.0 stable.