CAN bus anomalies with 6HC and 3HC

adammhaile

@chrishamm Interesting...
I've been using one of these microSD cards which is typical for me on the Pi and especially for one that is in a setup like this where "properly" shutting it down each time is not easy.

I noticed in the docs mention of an SD card speed test, which I ran but I'm thinking that it is only meant for a card mounted in the Duet, not the SBC... because... well, these are horrible numbers:

4/12/2022, 8:35:48 AM	M122 P104 S5
Testing SD card write speed...
4/12/2022, 8:36:26 AM	SD write speed for 5.0Mbyte file was 0.13Mbytes/sec
4/12/2022, 8:36:26 AM	Testing SD card read speed...
4/12/2022, 8:43:50 AM	SD read speed for 5.0Mbyte file was 0.01Mbytes/sec

As for CPU usage - Note: this is a Pi 4 w/ 4GB RAM. No overclock.

This is at machine idle - just on, no job running:

This is during the text at the bottom of a benchy - so tons of tiny moves:

This is a few seconds after the last, with an mjpg_streamer camera stream started:

chrishamm

@adammhaile The CPU usage looks OK but I agree the SD test is pretty disappointing. I've been using these SanDisk Extreme 64GB A2 cards and overwrote all of them countless times for DuetPi tests and they're still perfectly fine.

I'm still happy with the Samsung SSDs I have but I cannot say much about the quality of their microSD cards.

adammhaile

@chrishamm said in CAN bus anomalies with 6HC and 3HC:

The CPU usage looks OK but I agree the SD test is pretty disappointing.

I still am confused by those results - If I run a perf test from the Pi command line (using agnostic) I'm getting 45MB/s writes and 60+ reads.

@chrishamm said in CAN bus anomalies with 6HC and 3HC:

I've been using these SanDisk Extreme 64GB A2 cards

Ha! I ordered 2 of those this morning

Do you think I would be safe simply cloning my existing SD to thew new one or should I start from scratch?

Phaedrux

I would back up the configs and start with a fresh DuetPi image, at least for testing. If you have more customizations you wish to preserve cloning the card should be an option.

adammhaile

@phaedrux said in CAN bus anomalies with 6HC and 3HC:

I would back up the configs and start with a fresh DuetPi image, at least for testing. If you have more customizations you wish to preserve cloning the card should be an option.

Ok, will do that for now then.

adammhaile

@phaedrux @chrishamm @dc42
I've got to remove these boards and send them back to Filastruder - anything else you want me to try before I do that?

T3P3Tony

@adammhaile when so the new SD cards arrive? Would it be too disruptive to ask you to test with one of those.

I realise that you have been plugging away at this issue for a while so if you can't wait then I understand.

adammhaile

@t3p3tony said in CAN bus anomalies with 6HC and 3HC:

when so the new SD cards arrive? Would it be too disruptive to ask you to test with one of those.

No problem - they arrive today. I can likely give it a shot tonight.

adammhaile

@chrishamm @T3P3Tony I don't think I trust the M122 P104 SD card test.... I'm using the new recommended SD card I've I've tested on my desktop at over 140MB/s but when running the diagnostic speed test I get the same exact results as before.
Honestly the fact that it's the exact same speed every time makes me feel like it's a bottleneck elsewhere - likely in the diagnostics code.
Especially given that I'm consistently able to upload gcode files at ~15MB/s no problem.
Though.... maybe this is part of the problem... The 15MB/s upload is through DWC which would be direct to the Pi.
But the diagnostics SD write test is running from the 6HC control board itself - so maybe that bottleneck is the SPI bus and that's causing my problems?

Is there anywhere I could purchase a new 6HC ribbon cable? Wondering if I should replace that too - though can't find anything that I'm sure is correct.

It's late now but I will run some print tests in the morning to see if I can cause any other fails, before I pack up the boards to ship back to Filastruder.

Phaedrux

You could measure the continuity and resistance on the ribbon cable, that would tell us if it's acceptable or not.

chrishamm

@adammhaile is this with the new SanDisk card?

adammhaile

@chrishamm said in CAN bus anomalies with 6HC and 3HC:

is this with the new SanDisk card?

Yes. Same card recommended above.

chrishamm

@adammhaile Please check if the disconnects persist with the new card. If they do, I'll be happy to share a new firmware build that tells us whether the timeout is caused by the SBC or by Reprapfirmware. We've got another trace but I cannot comment on that one yet.

adammhaile

@chrishamm said in CAN bus anomalies with 6HC and 3HC:

Please check if the disconnects persist with the new card. If they do, I'll be happy to share a new firmware build that tells us whether the timeout is caused by the SBC or by Reprapfirmware. We've got another trace but I cannot comment on that one yet.

Sure - been running prints off this SD all morning. So far so good - but it was pretty random before so we'll see.
And by timeouts do you mean the SPI connection reset?

I'll keep putting this through it's paces either through tomorrow morning or until it fails again - then I'll remove and packup to send back.

adammhaile

Alright @chrishamm @Phaedrux @dc42 - had it printing since early this morning and now (8pm) it locked up while not printing... couldn't even run any commands to get diagnostics. I could talk to the Pi, but no comms with the controllers until I power cycled.
I'm going to get these boards taken out of the machine now to

T3P3Tony

@adammhaile thanks for confirming that. I hope the replacement sorts the issue.

adammhaile

@t3p3tony @chrishamm @Phaedrux
<sigh> Got the replacements, installed them, and all seemed to be going fine... but was just running a print and it stopped again mid-print. I unfortunately wasn't even able to view the duetcontrolserver log. I could try to run journalctl but it just never returned - and this was from the Pi terminal directly. I couldn't ssh into it.
I was able get this though, which is the streaming output of the CodeLogger -t executed -q

So, I'll admit - probably not something with the actual Duet boards - though I'm completely stumped as to what it could be.
Guess maybe I'll try swapping the Pi again - the fact that even SSH locks up is suspect to me. Implies that it's not the Duet failing... I guess?

I'm running mjpg-streamer and that gcode scroll on the display above from the SBC Pi - we've previously checked that the CPU usage is still low... but could it maybe still be one of those causing it? I've got other Duet 3 SBC machines that have a camera running off the same Pi without issue.

Phaedrux

Start eliminating extras until you find the smoking gun.

adammhaile

@phaedrux said in CAN bus anomalies with 6HC and 3HC:

Start eliminating extras until you find the smoking gun.

Yup - working on it. Re-running the same ~12 hour print every day, changing one thing each time.

chrishamm

@adammhaile Those M905 codes are usually a symptom of excessive load on the Pi - DCS sends them whenever a scheduled delay takes +5s longer than expected (the current acceptable maximum for a 4s delay is already set to 9s, so plenty of time) or when the datetime has been changed (hence the corresponding messages in the duetcontrolserver journal).

So it would be interesting to see what actually prevents DCS from getting computing power and/or IO access to linked libs. Just to exclude IO from/to the microSD card, you could temporarily copy the entire DSF directory to /tmp and run DCS from there (that is from the Pi's memory):

cp -r /opt/dsf/bin /tmp/dsf
sudo systemctl stop duetcontrolserver
/tmp/dsf/DuetControlServer

Just be aware that you'll have to keep the terminal where you run this open, else a potential print would be aborted. If the same disconnects persists, it must be related to the CPU, memory, or kernel. It would be interesting to see what happens then.