Can-o-Worms: Pi Shutdown vs Power Off

Danal

SUMMARY

I built a 'test rig' to attempt to quantify the operational risk of powering off a Pi without a shutdown command. It uses an Arduino pro mini and a relay to control power to a Raspberry Pi 3B+. It does do IO to the SD on a regular basis (see details next post).

As of this writing, it has powered off more than 1460 times with no corruption to the SD card or file systems. It is still cycling; I am posting because 1460 is equivalent to 4 times per day for one year.

MOTIVATION

I am not 100% clear in my motivation for doing this, and even less clear about posting the results on the Duet forum. It is clear that:

Some people, including me, are going to power cycle Pis. Other people are going to shutdown first.

I do not wish to "force" change on anyone; at the same time, I'd like to see some facts on the table for those who are still making their choice. In analyzing my motivations, I anticipate that at least some people will see the test as invalid. And the discussion around the test setup may become deeper than the shutdown/poweroff discussion itself. I believe the test setup is valid... see details next post... and I am very willing to discuss more detail or improvements or etc.

CONCLUSION

For me, the operational risk of powering of a Pi without a shutdown is acceptable.

I fully acknowledge that there is no "proof" here and that the SD could corrupt on the next power cycle. Always backup.

Danal

TEST SETUP

The Pi:

Raspberry Pi 3B+ About two years old.
SD card is Sandisk "edge" 8GB. Not sure where it came from. Nothing special.
OS is imaged from the RaspberryPi foundation web site, regular Raspbian
Standard install for ssh, wireless, etc.
Nothing special done to file systems or checking similar.
Also installed 'sqllite3' DB and a bunch of scripts that I use on a different Pi as a monitor.
Changed the scripts to generate fake data every 2 seconds
This will result in plenty of activity to the SD
This actual monitor generates transactions every 15 seconds to 5 minutes
Therfore, the 'fake' data writes to the SD 30x to several hundred x more than the real monitor.

There is also a script that scans the boot log, and finds the entry 'systemd-fsckd.service: Succeeded.' (or failed). This is the file system check that is built into Raspbian. No change to the default image. Once this entry is found, it is pushed to a CIFS (windows) network shared drive and appended to a log file. That is my main way of knowing how many boot cycles have occured. It also validates that MANY things on the PI/Card are working, not corrupted.

The power cycle:

Eventek KPS3010D benchtop power supply set to 12v.
Arduino micro pro running small custom sketch.
Sketch closes relay (power on) for two minutes, opens relay (power off) for 30 seconds, repeats.
Relays are ?songle? 10A at 50V DC
There are two relays so that + and - power wires are both cut.
The relays are driven from Arduino IO 3 and 4, via a FET
12V power goes from the relay to a "UBEC" that outputs 5.1V up to 3A
Power wires from the UBEC are soldered to the Pi on the back of the board, near the micro USB power connector (reused a prior setup)
There is also a button that signals the Arduino script to not cycle off, so that I can ssh into the Pi and fix anything, if required (so far, not)

This was re-used from a setup from years back, so it has extra stuff not relevant to this test; it was just a handy way to get it set up quickly:

Danal

Other thoughts on the test rig:

As stated over and over, this is "nothing special", every effort to make it a regular Pi Raspbian build.
It DOES do IO to the card. In fact, averages a lot more than the Pi in an idle printer will ever be writing.
Some people may ask for even more IO. This would be unrealistic. Very heavy IO shouldn't be done to an SD card; if your Pi is going to be doing tons of IIOPS, boot from an SD and read/write to a USB disk of some sort.

What the test is not:

It is NOT "proof". Neither in the formal or informal sense.
- For one thing, there is no control group
- For another, you cannot 'prove' a negative. Logic 101.
It is NOT designed to separate out events caused by the fact that SD cards WILL eventually fail from write activity
- To be clear, no such events have occurred
It is NOT based on any bench-marking tools. For example, the things I picked to do IO are completely arbitrary
It is NOT "deep scanning" the file systems.
- The definition of 'success' is 'the boot fsck is happy' and that the Pi 'operates normally over time'.
- That second one is the real key.
- If corruption occurs and is not fixed, it should get worse very quickly, resulting in an un-runnable system.
It is NOT certified or audited by anyone at all. You have to trust me that I'm not lying or witholding any part of the data.
It is NOT the way you would do it.
It is NOT intended to eliminate all risk of losing the SD contents. Always backup.

What the test is:

It is statistical in nature.
It is intended to be as realistic as possible.
It is intended to run enough times to quantify operational risk.
It is sitting there continuing to yank power every 2.5 minutes.

I don't know if I will run it to failure, or not. It is taking up a bit of bench space and a power supply.

Luke'sLaboratory

If you do come up with a result, I'd be curious to see what the MTTF is for a similar board, the Odrid C2, which can upgrade to using EMMC modules instead of sd cards.

chas2706

@Danal

Just for your info,

until recently upgrading one of my printers to Duet 3 + SBC I had a raspberry pi3 + usb camera on the old set up running motioneyeos.

Because the pi's are so cheap I never bothered to shutdown the pi in the correct way.

When i'd finished printing I would just power the printer off which would immediately cut power to the pi. (I fed it from a 5v buck converter hooked up to my printer psu).

I used this set up for approx. 2 - 3 years and I can confirm that I never had any issues.
The sd card has never corrupted.

In fact it is the same rpi3, sd card and motioneyos.img that is now running on my new duet 3 set up. (I have rpi4 as the SBC).

Luke'sLaboratory

@chas2706 brings up a good point. I don't think I've ever properly shut down my pi on any of my projects (reboots notwithstanding) that goes back 5+ years. All of my octoprint installs are all on the same power supply as my printers, and they'll hit probably a cycle a week or so.

My duet3-based system however gets power cycled on average twice a day.

gtj0

@Danal Out of curiosity, how much time elapses between the 12v cutout and the 5v dropping below the Pi's Vin minimum?

chas2706

@Luke-sLaboratory said in Can-o-Worms: Pi Shutdown vs Power Off:

brings up a good point. I don't think I've ever properly shut down my pi on any of my projects (reboots notwithstanding) that goes back 5+ years. All of my octoprint installs are all on the same power supply as my printers, and they'll hit probably a cycle a week or so.
My duet3-based system however gets power cycled on average twice a day.

Yes I can well believe it.

I have played around with raspberry pi's since they first came on the market.
Done loads of projects with them and used many different versions of os's and cannot recall anytime where the sd card(s) became corrupt upon power failure.

I also have a home made raspberry pi cctv system that runs 24/7 and has done for the last 6 years and has experienced some power failures but without issue.

deckingman

@Danal Please excuse my ignorance because my RPi although fitted, remains unconnected so I have zero experience of using the thing. I assume that the reason why you have gone to these lengths is because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" yes?

chas2706

@gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:

Out of curiosity, how much time elapses between the 12v cutout and the 5v dropping below the Pi's Vin minimum?

On my new set up I have a separate beefy 5v 10 amp psu to supply my 2 pi's .
They both now do a proper shutdown. I purchased a 5v relay module of which the contacts are connected to the 230v live to the system.
I have a rpi 4 as the SBC and a rpi 3 running motioneyeos for my webcam.

The relay input is connected to a spare gpio pin (pin18) on the SBC pi ( I got rid of the ribbon connector that stops you using the unused gpio pins and made my own 6 wire connector).
The relay board is fed from 3.3v on the pi and is active low.

On the SBC pi I set pin 18 as active low on start up and made a small script called "shutdown pi's" which is accessible from the pi start menu.
The script firstly connects to the rpi 3 via ssh and gives it the shutdown command then there is a deliberate time delay before the rpi 4 is given the shutdown command.

It all works a treat and I think it is down to having the rpi's fed from external (beefy) psu because both rpi's have more than enough time to shut down completely before the external psu loses all of its power!

DanS79

@deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

Please excuse my ignorance because my RPi although fitted, remains unconnected so I have zero experience of using the thing. I assume that the reason why you have gone to these lengths is because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" yes?

A Pi really isn't really any different than your desktop. If you suddenly loose power, you can corrupt the file system under the right conditions. Depending on how your system is configured and what its doing at the time of power loss, your odds of corruption could be anywhere from guaranteed to 1 in a trillion.

It really depends on how much you care about potentially loosing data.

For example if not loosing data is important to you, you could do something like this.
https://www.youtube.com/watch?v=m20o3P-S5s8

Admittedly never had a Pi corrupt a card, but plenty of other linux based systems packing in due to power loss, so I'm not about to call it a theoretical problem or stop shutting down Pi's cleanly, if I couldn't be bothered to run poweroff I'd just leave the thing running.

If the Pi and accompanying software were sufficiently stable you could simply run sudo raspi-config and enable read-only filesystem. But with new updates weekly or every other day that becomes cumbersome if not integrated in the upgrade scripts.

In general the whole Pi setup needs hardening and I'm sure it will come, but I also suspect the team needs to make it usable first.

Re: EMMC; ask Tesla what they think; although their issues are more related to write (erase) operations than shutdowns.

Phaedrux

I see your reckless Pi shutdowns and raise you about 50 Atom CPU based windows 10 mini PCs that get unceremoniously power killed on a regular basis (power outages, regular maintenance, etc) for the past 3 years and have not had any OS corruption at all. They use an onboard eMMC flash storage device (basically an SD card soldered to the mobo). Modern file systems and flash storage devices just aren't as susceptible to power loss corruption as older filesystems on spinning magnetic storage.

deckingman

@DanS79 Yes, I get that. But it doesn't answer my question.

@Danal has gone to the length of building a test rig and after much evaluation has arrived at the conclusion (for him personally) quote "For me, the operational risk of powering of a Pi without a shutdown is acceptable.".

Now @Danal doesn't strike me as being the sort of person who builds test rigs unless there is some reason for doing so. In which case, I concluded that there must be a reason why he prefers to "pull the plug" rather than doing a controlled shutdown. So my question remains. - "Is it because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" ?

Danal

@gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:

@Danal Out of curiosity, how much time elapses between the 12v cutout and the 5v dropping below the Pi's Vin minimum?

I will see if there is a reasonable way to measure that. For example, if my dual channel scope will go slowly enough to trigger on the 12V and also draw a trace for the 5V.

If you have any thoughts on how to measure...?

Phaedrux

@deckingman I think the downsides come down to complexity and time. Since the pi isn't always using a display and keyboard and mouse it can take time to get it shut down cleanly because you might have to remote into it on your phone or other PC or connect a display and mouse. Plus it also means waiting for the system to shut down. Or you could maybe wire up some sort of trigger relay to shut it down cleanly with a button press, but you still have to wait for that.

BUT if there is really negligible risk to simply killing the power, you can just unplug it or flip a power switch, or have it wired alongside the Duet which gets killed when power is removed to the printer, and not really worry about it.

@deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

So my question remains. - "Is it because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" ?

no, there are many ways to achieve controlled shutdown - but there are tradeoffs with respect to security and at the end of the day it doesn't protect against power loss (unless you add battery backup to the mix)

garyd9

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

Modern file systems and flash storage devices just aren't as susceptible to power loss corruption as older filesystems on spinning magnetic storage.

While I tend to agree to some extent, any reasonable general purpose caching filesystem can be corrupted from a power loss, regardless of the backing storage technology. If power is lost in the middle of a write, corruption can occur. A filesystem check on a filesystem with a journal often will "repair" the filesystem, but without knowing the intent of the programs writing data, it might be incapable of properly repairing the data. (A classic example is a database with bad indexes due to a power loss between data and index writes.)

I'll point out the duet wiki page of How to Destroy your Duet 2. In that document, there's a clear disclaimer of "We can't guarantee that these techniques will not destroy your Duet 2" and " doing anything described here will invalidate your warranty!"

The same goes for unplugging your raspberryPi without shutting down the filesystems cleanly. Just because it "probably" won't cause you headaches, or just because it didn't cause problems X times before, doesn't mean it's safe or a good idea.

Danal

@deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

@Danal Please excuse my ignorance because my RPi although fitted, remains unconnected so I have zero experience of using the thing. I assume that the reason why you have gone to these lengths is because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" yes?

Two very separate reasons I built the test rig.

When this gets discussed (here and other forums), and I say 'I just power off', some people react as though my Pi is in control of a nuclear bomber overflying their neighborhood. Seriously, you'd be amazed at how nasty some people have been that I might be misleading noobys or something. This forum is pretty respectful, many are not.
Reason two is really a whole laundry list:

I'm just lazy.
I have more Pi's than just the one in the printer. Some of them are easy to access for a "sudo shutdown". Many of them are not.
I'm just lazy.
The need for shutdown of journaling file systems has struck me as BS for decades. I recognize that some system really do need to on a "pragmatic" basis (high risk of corruption)... I've always believed philosophically that was unacceptable design on the part of the OS and Filesystem designers. Foisting a file system on the public that is beyond its own provided "check at power up" is almost criminally negligent and is absolutely technically arrogant.
I'm just lazy.
And I could have been wrong. This test rig could have demonstrated that corruption is 1:10 or 1:100 odds, not the 1:thousands that it seems to show.

And mostly, I'm just lazy.

Danal

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

BUT if there is really negligible risk to simply killing the power, you can just unplug it or flip a power switch, or have it wired alongside the Duet which gets killed when power is removed to the printer, and not really worry about it.

Exactly.

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

and not really worry about it.

read only file system gives peace of mind, statistics not so much

Phaedrux

@garyd9 said in Can-o-Worms: Pi Shutdown vs Power Off:

The same goes for unplugging your raspberryPi without shutting down the filesystems cleanly. Just because it "probably" won't cause you headaches, or just because it didn't cause problems X times before, doesn't mean it's safe or a good idea.

Totally agree, and that's where the risk must be weighed. How much time will be wasted in the case of a failure? How can that time be mitigated with backups, etc?

This is where I have a hard time because I haven't yet been able to cause the failure situation in normal usage, so I'm not really sure how much pain would go into a recovery. Worst case scenario for a Pi would be maybe replacing the SD card AND the pi itself. So that's maybe 50$ financial hit, plus the time to download an up to date pi image and install it on the fresh SD card, update the system, restore any backups or do a fresh install of whatever software.

In the case of the 50 mini pcs I mentioned, the risk was deemed acceptable, because the cost and complexity of mitigation was far too high to justify. If a mini PC dies it simply gets replaced with a spare. At 200$ each it's far more than a pi, but it's so rare, and the mitigation impossible (battery backup is out of the question in this locale), that it's an accepted risk.

If anyone can show me a reliable way to KILL a pi or SD card (without actual negligence/malfeasance) I'd love to see it.

garyd9

Not sure what kind of writes your doing in your test, but perhaps try something a bit more annoying to filesystems: zip. (might need to "apt install zip")

The reason is that the headers in a zip archive have to be updated after compressed data is written to the file, so there's a chance that the archive header is out of date with respect to the data in the archive, rendering the entire archive corrupted (even if the filesystem isn't corrupted.)

Just run "zip test.zip /usr/bin/*" and on OS startup, test the zip for integrity.

garyd9

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

Totally agree, and that's where the risk must be weighed. How much time will be wasted in the case of a failure? How can that time be mitigated with backups, etc?

I'm not familiar with what DSF/DSC is doing behind the scenes, but a question is what it might be writing to the filesystem. Anything in /var/log can be redirected to a tempfs (it'd be lost on power failure.) If that's all the fs writing that that the duet stuff does (other than writing gcode files in direct response to user interaction), then it should be reasonably safe. I'd probably turn off write caching on the file system as well (with nearly zero penalty if logs are all going to ramdisk.)

deckingman

@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:

....................Seriously, you'd be amazed at how nasty some people have been that I might be misleading noobys or something. This forum is pretty respectful, many are not.

Ohh, I'm not amazed - I have my own blog and YouTube Channel so I know how true that is.

And mostly, I'm just lazy.

Thanks for your honesty. I'm even more lazy - maybe that's why I'm sticking to stand alone "RPi-less" operation. If I don't start it up, I don't have to worry about shutting it down