Can-o-Worms: Pi Shutdown vs Power Off

DanS79 · 21 Apr 2020, 20:02

@deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

Please excuse my ignorance because my RPi although fitted, remains unconnected so I have zero experience of using the thing. I assume that the reason why you have gone to these lengths is because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" yes?

A Pi really isn't really any different than your desktop. If you suddenly loose power, you can corrupt the file system under the right conditions. Depending on how your system is configured and what its doing at the time of power loss, your odds of corruption could be anywhere from guaranteed to 1 in a trillion.

It really depends on how much you care about potentially loosing data.

For example if not loosing data is important to you, you could do something like this.
https://www.youtube.com/watch?v=m20o3P-S5s8

A Former User · 21 Apr 2020, 20:10

Admittedly never had a Pi corrupt a card, but plenty of other linux based systems packing in due to power loss, so I'm not about to call it a theoretical problem or stop shutting down Pi's cleanly, if I couldn't be bothered to run poweroff I'd just leave the thing running.

If the Pi and accompanying software were sufficiently stable you could simply run sudo raspi-config and enable read-only filesystem. But with new updates weekly or every other day that becomes cumbersome if not integrated in the upgrade scripts.

In general the whole Pi setup needs hardening and I'm sure it will come, but I also suspect the team needs to make it usable first.

Re: EMMC; ask Tesla what they think; although their issues are more related to write (erase) operations than shutdowns.

Phaedrux · 21 Apr 2020, 20:11

I see your reckless Pi shutdowns and raise you about 50 Atom CPU based windows 10 mini PCs that get unceremoniously power killed on a regular basis (power outages, regular maintenance, etc) for the past 3 years and have not had any OS corruption at all. They use an onboard eMMC flash storage device (basically an SD card soldered to the mobo). Modern file systems and flash storage devices just aren't as susceptible to power loss corruption as older filesystems on spinning magnetic storage.

deckingman · 21 Apr 2020, 20:20

@DanS79 Yes, I get that. But it doesn't answer my question.

@Danal has gone to the length of building a test rig and after much evaluation has arrived at the conclusion (for him personally) quote "For me, the operational risk of powering of a Pi without a shutdown is acceptable.".

Now @Danal doesn't strike me as being the sort of person who builds test rigs unless there is some reason for doing so. In which case, I concluded that there must be a reason why he prefers to "pull the plug" rather than doing a controlled shutdown. So my question remains. - "Is it because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" ?

Danal · 21 Apr 2020, 20:20

@gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:

@Danal Out of curiosity, how much time elapses between the 12v cutout and the 5v dropping below the Pi's Vin minimum?

I will see if there is a reasonable way to measure that. For example, if my dual channel scope will go slowly enough to trigger on the 12V and also draw a trace for the 5V.

If you have any thoughts on how to measure...?

Phaedrux · 21 Apr 2020, 20:28

@deckingman I think the downsides come down to complexity and time. Since the pi isn't always using a display and keyboard and mouse it can take time to get it shut down cleanly because you might have to remote into it on your phone or other PC or connect a display and mouse. Plus it also means waiting for the system to shut down. Or you could maybe wire up some sort of trigger relay to shut it down cleanly with a button press, but you still have to wait for that.

BUT if there is really negligible risk to simply killing the power, you can just unplug it or flip a power switch, or have it wired alongside the Duet which gets killed when power is removed to the printer, and not really worry about it.

A Former User · 21 Apr 2020, 20:29

@deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

So my question remains. - "Is it because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" ?

no, there are many ways to achieve controlled shutdown - but there are tradeoffs with respect to security and at the end of the day it doesn't protect against power loss (unless you add battery backup to the mix)

garyd9 · 21 Apr 2020, 20:30

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

Modern file systems and flash storage devices just aren't as susceptible to power loss corruption as older filesystems on spinning magnetic storage.

While I tend to agree to some extent, any reasonable general purpose caching filesystem can be corrupted from a power loss, regardless of the backing storage technology. If power is lost in the middle of a write, corruption can occur. A filesystem check on a filesystem with a journal often will "repair" the filesystem, but without knowing the intent of the programs writing data, it might be incapable of properly repairing the data. (A classic example is a database with bad indexes due to a power loss between data and index writes.)

I'll point out the duet wiki page of How to Destroy your Duet 2. In that document, there's a clear disclaimer of "We can't guarantee that these techniques will not destroy your Duet 2" and " doing anything described here will invalidate your warranty!"

The same goes for unplugging your raspberryPi without shutting down the filesystems cleanly. Just because it "probably" won't cause you headaches, or just because it didn't cause problems X times before, doesn't mean it's safe or a good idea.

Danal · 21 Apr 2020, 20:31

@deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

@Danal Please excuse my ignorance because my RPi although fitted, remains unconnected so I have zero experience of using the thing. I assume that the reason why you have gone to these lengths is because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" yes?

Two very separate reasons I built the test rig.

When this gets discussed (here and other forums), and I say 'I just power off', some people react as though my Pi is in control of a nuclear bomber overflying their neighborhood. Seriously, you'd be amazed at how nasty some people have been that I might be misleading noobys or something. This forum is pretty respectful, many are not.
Reason two is really a whole laundry list:

I'm just lazy.
I have more Pi's than just the one in the printer. Some of them are easy to access for a "sudo shutdown". Many of them are not.
I'm just lazy.
The need for shutdown of journaling file systems has struck me as BS for decades. I recognize that some system really do need to on a "pragmatic" basis (high risk of corruption)... I've always believed philosophically that was unacceptable design on the part of the OS and Filesystem designers. Foisting a file system on the public that is beyond its own provided "check at power up" is almost criminally negligent and is absolutely technically arrogant.
I'm just lazy.
And I could have been wrong. This test rig could have demonstrated that corruption is 1:10 or 1:100 odds, not the 1:thousands that it seems to show.

And mostly, I'm just lazy.

Danal · 21 Apr 2020, 20:32

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

BUT if there is really negligible risk to simply killing the power, you can just unplug it or flip a power switch, or have it wired alongside the Duet which gets killed when power is removed to the printer, and not really worry about it.

Exactly.

A Former User · 21 Apr 2020, 20:32

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

and not really worry about it.

read only file system gives peace of mind, statistics not so much

Phaedrux · 21 Apr 2020, 20:40

@garyd9 said in Can-o-Worms: Pi Shutdown vs Power Off:

The same goes for unplugging your raspberryPi without shutting down the filesystems cleanly. Just because it "probably" won't cause you headaches, or just because it didn't cause problems X times before, doesn't mean it's safe or a good idea.

Totally agree, and that's where the risk must be weighed. How much time will be wasted in the case of a failure? How can that time be mitigated with backups, etc?

This is where I have a hard time because I haven't yet been able to cause the failure situation in normal usage, so I'm not really sure how much pain would go into a recovery. Worst case scenario for a Pi would be maybe replacing the SD card AND the pi itself. So that's maybe 50$ financial hit, plus the time to download an up to date pi image and install it on the fresh SD card, update the system, restore any backups or do a fresh install of whatever software.

In the case of the 50 mini pcs I mentioned, the risk was deemed acceptable, because the cost and complexity of mitigation was far too high to justify. If a mini PC dies it simply gets replaced with a spare. At 200$ each it's far more than a pi, but it's so rare, and the mitigation impossible (battery backup is out of the question in this locale), that it's an accepted risk.

If anyone can show me a reliable way to KILL a pi or SD card (without actual negligence/malfeasance) I'd love to see it.

garyd9 · 21 Apr 2020, 20:40

Not sure what kind of writes your doing in your test, but perhaps try something a bit more annoying to filesystems: zip. (might need to "apt install zip")

The reason is that the headers in a zip archive have to be updated after compressed data is written to the file, so there's a chance that the archive header is out of date with respect to the data in the archive, rendering the entire archive corrupted (even if the filesystem isn't corrupted.)

Just run "zip test.zip /usr/bin/*" and on OS startup, test the zip for integrity.

garyd9 · 21 Apr 2020, 20:45

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

Totally agree, and that's where the risk must be weighed. How much time will be wasted in the case of a failure? How can that time be mitigated with backups, etc?

I'm not familiar with what DSF/DSC is doing behind the scenes, but a question is what it might be writing to the filesystem. Anything in /var/log can be redirected to a tempfs (it'd be lost on power failure.) If that's all the fs writing that that the duet stuff does (other than writing gcode files in direct response to user interaction), then it should be reasonably safe. I'd probably turn off write caching on the file system as well (with nearly zero penalty if logs are all going to ramdisk.)

deckingman · 21 Apr 2020, 20:47

@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:

....................Seriously, you'd be amazed at how nasty some people have been that I might be misleading noobys or something. This forum is pretty respectful, many are not.

Ohh, I'm not amazed - I have my own blog and YouTube Channel so I know how true that is.

And mostly, I'm just lazy.

Thanks for your honesty. I'm even more lazy - maybe that's why I'm sticking to stand alone "RPi-less" operation. If I don't start it up, I don't have to worry about shutting it down

Danal · 21 Apr 2020, 20:51

I'm updating a SQLLITE3 database. Fairly simple structure... but... to your point, the DB engine is writing all over a file that the file systems just sees as a bunch of blocks. And those relationships can break.

Just checked. DB Healthy.

A Former User · 21 Apr 2020, 21:00

@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

Totally agree, and that's where the risk must be weighed. How much time will be wasted in the case of a failure? How can that time be mitigated with backups, etc?

I'm not gonna name names, but I think most of us can think of a good handful that would struggle for weeks with such a scenario; which beggs the question why advocate something that is probably safe instead of implementing something that is safe? And accepting that the recommendation while we wait for a safe solution should be to safely shut down the system - anyone can ignore it at their own risk.

Phaedrux · 21 Apr 2020, 21:06

@bearer said in Can-o-Worms: Pi Shutdown vs Power Off:

And accepting that the recommendation while we wait for a safe solution should be to safely shut down the system - anyone can ignore it at their own risk.

Yes I totally get your point. A clean shutdown is the safest bet and best practice. But it's also good to know that if power is lost unexpectedly that it's not guaranteed catastrophe.

gloomyandy · 21 Apr 2020, 21:09

@bearer If you are going to use a read only pi filesystem for the sbc, where are you going to upload gcode files to? Do you intend to use a 2nd disk (network share?) of some sort?

gtj0 · 21 Apr 2020, 21:10

@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:

@gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:

@Danal Out of curiosity, how much time elapses between the 12v cutout and the 5v dropping below the Pi's Vin minimum?

I will see if there is a reasonable way to measure that. For example, if my dual channel scope will go slowly enough to trigger on the 12V and also draw a trace for the 5V.

If you have any thoughts on how to measure...?

Connect 12V to channel 1 and 5v to channel 2. Set the scope it to trigger on channel 1 falling edge.