CAN bus anomalies with 6HC and 3HC

adammhaile

@phaedrux @chrishamm @dc42
I've got to remove these boards and send them back to Filastruder - anything else you want me to try before I do that?

T3P3Tony

@adammhaile when so the new SD cards arrive? Would it be too disruptive to ask you to test with one of those.

I realise that you have been plugging away at this issue for a while so if you can't wait then I understand.

adammhaile

@t3p3tony said in CAN bus anomalies with 6HC and 3HC:

when so the new SD cards arrive? Would it be too disruptive to ask you to test with one of those.

No problem - they arrive today. I can likely give it a shot tonight.

adammhaile

@chrishamm @T3P3Tony I don't think I trust the M122 P104 SD card test.... I'm using the new recommended SD card I've I've tested on my desktop at over 140MB/s but when running the diagnostic speed test I get the same exact results as before.
Honestly the fact that it's the exact same speed every time makes me feel like it's a bottleneck elsewhere - likely in the diagnostics code.
Especially given that I'm consistently able to upload gcode files at ~15MB/s no problem.
Though.... maybe this is part of the problem... The 15MB/s upload is through DWC which would be direct to the Pi.
But the diagnostics SD write test is running from the 6HC control board itself - so maybe that bottleneck is the SPI bus and that's causing my problems?

Is there anywhere I could purchase a new 6HC ribbon cable? Wondering if I should replace that too - though can't find anything that I'm sure is correct.

It's late now but I will run some print tests in the morning to see if I can cause any other fails, before I pack up the boards to ship back to Filastruder.

Phaedrux

You could measure the continuity and resistance on the ribbon cable, that would tell us if it's acceptable or not.

chrishamm

@adammhaile is this with the new SanDisk card?

adammhaile

@chrishamm said in CAN bus anomalies with 6HC and 3HC:

is this with the new SanDisk card?

Yes. Same card recommended above.

chrishamm

@adammhaile Please check if the disconnects persist with the new card. If they do, I'll be happy to share a new firmware build that tells us whether the timeout is caused by the SBC or by Reprapfirmware. We've got another trace but I cannot comment on that one yet.

adammhaile

@chrishamm said in CAN bus anomalies with 6HC and 3HC:

Please check if the disconnects persist with the new card. If they do, I'll be happy to share a new firmware build that tells us whether the timeout is caused by the SBC or by Reprapfirmware. We've got another trace but I cannot comment on that one yet.

Sure - been running prints off this SD all morning. So far so good - but it was pretty random before so we'll see.
And by timeouts do you mean the SPI connection reset?

I'll keep putting this through it's paces either through tomorrow morning or until it fails again - then I'll remove and packup to send back.

adammhaile

Alright @chrishamm @Phaedrux @dc42 - had it printing since early this morning and now (8pm) it locked up while not printing... couldn't even run any commands to get diagnostics. I could talk to the Pi, but no comms with the controllers until I power cycled.
I'm going to get these boards taken out of the machine now to

T3P3Tony

@adammhaile thanks for confirming that. I hope the replacement sorts the issue.

adammhaile

@t3p3tony @chrishamm @Phaedrux
<sigh> Got the replacements, installed them, and all seemed to be going fine... but was just running a print and it stopped again mid-print. I unfortunately wasn't even able to view the duetcontrolserver log. I could try to run journalctl but it just never returned - and this was from the Pi terminal directly. I couldn't ssh into it.
I was able get this though, which is the streaming output of the CodeLogger -t executed -q

So, I'll admit - probably not something with the actual Duet boards - though I'm completely stumped as to what it could be.
Guess maybe I'll try swapping the Pi again - the fact that even SSH locks up is suspect to me. Implies that it's not the Duet failing... I guess?

I'm running mjpg-streamer and that gcode scroll on the display above from the SBC Pi - we've previously checked that the CPU usage is still low... but could it maybe still be one of those causing it? I've got other Duet 3 SBC machines that have a camera running off the same Pi without issue.

Phaedrux

Start eliminating extras until you find the smoking gun.

adammhaile

@phaedrux said in CAN bus anomalies with 6HC and 3HC:

Start eliminating extras until you find the smoking gun.

Yup - working on it. Re-running the same ~12 hour print every day, changing one thing each time.

chrishamm

@adammhaile Those M905 codes are usually a symptom of excessive load on the Pi - DCS sends them whenever a scheduled delay takes +5s longer than expected (the current acceptable maximum for a 4s delay is already set to 9s, so plenty of time) or when the datetime has been changed (hence the corresponding messages in the duetcontrolserver journal).

So it would be interesting to see what actually prevents DCS from getting computing power and/or IO access to linked libs. Just to exclude IO from/to the microSD card, you could temporarily copy the entire DSF directory to /tmp and run DCS from there (that is from the Pi's memory):

cp -r /opt/dsf/bin /tmp/dsf
sudo systemctl stop duetcontrolserver
/tmp/dsf/DuetControlServer

Just be aware that you'll have to keep the terminal where you run this open, else a potential print would be aborted. If the same disconnects persists, it must be related to the CPU, memory, or kernel. It would be interesting to see what happens then.

adammhaile

@chrishamm Interesting - will keep that in mind.
Strangely the SD card speed test is exactly the same with the new Duet boards, new Pi, and new faster SD card (the one recommended above). So I'd say that test is a red herring. I'm currently seeing 150+ MB/s reads on this card from the Pi.

Will test with DCS in RAM if my current batch of tests fail to rule anything out.

adammhaile

@t3p3tony @chrishamm @Phaedrux @dc42
Update: I've been stress testing the machine after swapping the Pi again and tweaking some of the wiring runs to help avoid signal noise. The last 4 days I've started a 12 hour print every morning so that I could have it printing as long as possible without having to leave it running while I'm asleep.

The first 3 days went beautifully but today I got another of the same failures.
However, this time was a little different...

Previously the SBC Pi was running a dhcp server to provide an IP for a direct ethernet connected Pi running android and a touch screen DWC interface.
I ditched that when I rebuilt the DuetPi Lite image and instead installed a small WiFi/Ethernet router. So now the SBC Pi and DWC Android Pi are wired to that and I can connect to the machine over it's own WiFi access point (this is so I can get to it at MRRF ).

The advantage of this was that previously when it would fail, I would never be able to connect to the Pi over SSH which limited my ability to diagnose. But this time I could hit its ethernet connection (which, unlike the WiFi, doesn't go down when the failure occurs it would seem) via the WiFi router I added.

So that led me to digging into various system logs and eventually dmesg, where I found a stream of errors, such as these:

[31055.833034] ieee80211 phy0: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[31062.393030] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[31064.953033] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[31064.953060] ieee80211 phy0: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[31071.513043] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110

And it was those that led me to this Duet Forum post:

https://forum.duet3d.com/topic/23889/pi4-network-disconnect

Now, the crazy thing is: they were seeing exactly what I was.

SPI comms reset
WiFi dropping
Print stopping midway through and DWC showing it was done
Using a Pi 4 with a Duet 3 (mini, but still)

Fortunately, they seem to have a solution! In that post they recommended not using 5Ghz networks and disabling the wifi power management features. From further research it seems that WiFi with 5GHz is somewhat of an ongoing issue with the Pi 4 and it will sometimes just kill WiFi when it shouldn't.... Now, why that seems to affect SPI, I have no idea. But they seem to be related...

While I could disable power saving on WiFi, the 2.4GHz network part was a problem for silly reasons with my home WiFi access point.
So, for now I've switched to only using ethernet and have connected a separate USB ethernet adapter for the connection between the SBC Pi and the DWC Android Pi.

Whether or not this will also fix the problem for me... time will tell. I'll be back to long test prints tomorrow morning and let you know how it goes in a few days.
But so far this seems promising given they were seeing pretty identical symptoms. My fallback at this point is tearing a Pi 3B+ out of something else in my house and using that. I have another Duet 3 machine with a 3B+ and it's been rock solid.

chrishamm

@adammhaile Thanks for the info, that's very good to know. When I build the next DuetPi image, I'll disable WiFi power saving by default.

adammhaile

@chrishamm said in CAN bus anomalies with 6HC and 3HC:

@adammhaile Thanks for the info, that's very good to know. When I build the next DuetPi image, I'll disable WiFi power saving by default.

Awesome.
Silly request while you are in there... is there any chance you would considering setting up the dsf user to have a home directory and default shell so that you can log into it?
With complex configurations, such as mine, it's extremely helpful to be able to ssh directly as that user. I currently use usermod to do this on each Duet SBC I have and then will use Visual Studio Code's remote features to directly connect to the sd directory contents over SSH. This has also allowed me to keep the entire config backed up on GitHub

adammhaile

@chrishamm @dc42 @Phaedrux @T3P3Tony
Just wanted to follow up on this and let you know that, at this point, I'm quite confident we found the problem. So, thank you all for sticking with me through all the ups and downs of solving this. It's been a crazy ride for sure.
And hopefully with the patches made to the Pi image this won't be a problem for people in the future who want to use a Pi4.