Mid print hang 3.4.0b2, duet3, sbc
-
Here is the DWC output when the stall happens. The event a 1:13 was during a print, the event at 1:16 was after.
-
Can you set up some additional monitoring on the pi?
https://duet3d.dozuki.com/Wiki/Getting_Started_With_Duet_3#Section_Monitoring_optional
-
@phaedrux started DCS with debugging and imediately saw this: I'm going to run a print and see what happens.
[warn] Bad data CRC32 (expected 0x7122d0a5, got 0x68c5ab97) [warn] Bad data CRC32 (expected 0xc31600e2, got 0xe7708562) [warn] Bad data CRC32 (expected 0xc8a7c26b, got 0x5b1340ab) [warn] Bad data CRC32 (expected 0x312c52a0, got 0x2c1f8ef5) [warn] Bad data CRC32 (expected 0xd1710152, got 0x1c81bcd5)edit: couldn't get into the web interface. Got this:
System.OperationCanceledException: Board is not available (no header) at DuetControlServer.SPI.DataTransfer.ExchangeHeader() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/DataTransfer.cs:line 1436 at DuetControlServer.SPI.DataTransfer.PerformFullTransfer(Boolean connecting) in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/DataTransfer.cs:line 194 [info] Connection to Duet established [debug] Updated key spindles [debug] Requesting update of key state, seq 0 -> 1 [debug] Updated key state [debug] Requesting update of key tools, seq 0 -> 5 [debug] Updated key tools [debug] Requesting update of key volumes, seq 0 -> 0 [debug] Updated key volumes [debug] Requesting update of key boards, seq 0 -> 1087 [debug] Updated key boards [debug] Requesting update of key directories, seq 0 -> 0 [debug] Updated key directories [debug] Requesting update of key fans, seq 0 -> 7 [debug] Updated key fans [debug] Requesting update of key global, seq 0 -> 0 [debug] Updated key global [debug] Requesting update of key heat, seq 0 -> 10 [debug] Updated key heat [debug] Requesting update of key inputs, seq 0 -> 25 [debug] Updated key inputs [debug] Requesting update of key job, seq 0 -> 15 [warn] Bad data CRC32 (expected 0x3534c24d, got 0xc6f735d8) [debug] Updated key job [debug] Requesting update of key move, seq 0 -> 54 [warn] Bad data CRC32 (expected 0x390f64ca, got 0xffa85249) [debug] Updated key move [debug] Requesting update of key network, seq 0 -> 3 [debug] Updated key network [debug] Requesting update of key sensors, seq 0 -> 8 [debug] Updated key sensors [warn] Bad data CRC32 (expected 0x4c04f1ba, got 0x11d51962) [warn] Bad data CRC32 (expected 0x4bd6bb59, got 0x5ca43599) [warn] Bad data CRC32 (expected 0x4bd6bb59, got 0x01651ad2) [warn] Bad data CRC32 (expected 0xe75e751f, got 0x20b97c2f) [warn] Bad data CRC32 (expected 0x0259194f, got 0x1dc9c3ca) [warn] Bad data CRC32 (expected 0x59691b91, got 0xd61c979f) [fatal] Abnormal program termination [fatal] SPI task faulted System.Exception: RepRapFirmware refused message format at DuetControlServer.SPI.DataTransfer.ExchangeHeader() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/DataTransfer.cs:line 1540 at DuetControlServer.SPI.DataTransfer.PerformFullTransfer(Boolean connecting) in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/DataTransfer.cs:line 194 at DuetControlServer.SPI.Interface.Run() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/Interface.cs:line 1026 at DuetControlServer.Utility.PriorityThreadRunner.<>c__DisplayClass0_0.<Start>b__0() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/Utility/PriorityThreadRunner.cs:line 25 [fatal] SPI task faulted System.Exception: RepRapFirmware refused message format at DuetControlServer.SPI.DataTransfer.ExchangeHeader() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/DataTransfer.cs:line 1540 at DuetControlServer.SPI.DataTransfer.PerformFullTransfer(Boolean connecting) in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/DataTransfer.cs:line 194 at DuetControlServer.SPI.Interface.Run() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/Interface.cs:line 1026 at DuetControlServer.Utility.PriorityThreadRunner.<>c__DisplayClass0_0.<Start>b__0() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/Utility/PriorityThreadRunner.cs:line 25 [debug] Update task terminated [debug] IPC task terminated [debug] Job task terminated [debug] Periodic updater task terminated [info] Application has shut down -
Try a new SD card with a fresh duetpi image.
-
@phaedrux it seems turning on the berd air pump is causing the problem. Several times now I've switched on the berd air pump just to have the disconnect happen shortly after. I don't have a flyback diode on the pump, and originally I've had it connected to out3, but I tried moving it to out9 which should have an onboard flyback diode, and even now I have moved it on an external mosfet board with the machine still stopping once the pump turns on.
-
@zakm0n It sounds a lot like the SPI communication is interrupted by external sources. You may want to try to move other cables away from the SBC cable, replace it with a shorter one (if possible), or reduce the SPI frequency in
/opt/dsf/conf/config.json.Duet software engineer
-
@chrishamm I tried lowering the SPI frequency to half it's default. The SBC cable is run underneath the duet, so it should be well protected. I'm still getting disconnects because of SPI interruptions.
-
@chrishamm So, I have the chassis of the printer bonded to the mains earth, and after running a grounding wire to the body of the berd air pump, I'm now able to get about 3 hours into a print, but I'm still getting disconnects somewhere around the 3.5-4 hour mark into a print. Any other grounding you might suggest to get rid of noise? The SBC cable is run underneath of the Duet 3 and there's not really any way to further isolate it from the wiring of the machine. I'm at a loss here, and I'm considering ditching the Duet3 for something else going forward, as my personal machines with Klipper have absolutely never had such an issue, even in much less ideal setups.
-
@zakm0n said in Mid print hang 3.4.0b2, duet3, sbc:
The SBC cable is run underneath of the Duet 3 and there's not really any way to further isolate it from the wiring of the machine.
Foil tape as shielding?
-
@phaedrux I could try that, but I'll have to re-route the cable from behind the board. Foil tape has a way of shorting out things, doesn't it? I just can't understand how I'm seemingly the only one to ever have this problem.
-
@zakm0n said in Mid print hang 3.4.0b2, duet3, sbc:
Foil tape has a way of shorting out things, doesn't it?
A layer of non-conductive tape on top? There are purpose made shielded ribbon cables as well.
@zakm0n said in Mid print hang 3.4.0b2, duet3, sbc:
I just can't understand how I'm seemingly the only one to ever have this problem.
There are maybe a few other cases of similar interference I can think of, but it's usually been with a Duex ribbon cable or Paneldue picking up some noise.
The fact you're getting an improvement by increasing the grounding is promising though.
-
@zakm0n I grab Vin and GND for the Pi's buck converter relatively close from the Duet's Vin power connector in order to minimise potential interference.
I've been printing A LOT in SBC mode (countless prints > 7h) and I have not observed any connection drops on my setup. Do you get CRC errors when you bend or slightly twist the ribbon cable? If yes, try replacing it - a bad cable is the most plausible reason for your connection drops.
-
@chrishamm The PI has an external wall wart currently. The ribbon cable isn't in a place where it can be moved easily, so that hasn't been something I can test. Adding the grounding strap to the Berd pump got me from 10 minutes to 4 hours though, so I'm thinking grounding and shielding is where I should be focusing.
-
@phaedrux So, I managed to get the ribbon cable wrapped in metal foil tape and successfully printed a 6 hour print yesterday. I tried starting a print today and only made 10 minutes. It's like all of the gains I made by grounding and such just disappeared. It seems like this time there were a bunch of rapid connects and disconnects in the console on my paneldue.
-
And just as a verification if you have the berd air completely off the prints sill consistently complete without issue?
