Can-o-Worms: Pi Shutdown vs Power Off
-
@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:
SDs fail. Whether shutdown or not.
And power loss can happen even with a UPS. (accidentally unplugging it, battery dying, etc)
-
@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:
@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:
Huh? What mechanism increases or decreases the odds of damage to a Pi (not the SD, the Pi) if it is powered off before or after a Raspbian shutdown command?
Exactly. But I just chose that as an absolute worst case total loss situation on one side of the balance.
Huh again?
It seems we just agreed there is absolutely zero change in the odds of losing or not losing the hardware with shutdown or no shutdown. Therefore, hardware loss events do not enter into this discussion.
Yes? No?
Delta / Kossel printer fanatic
-
@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:
@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:
@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:
Huh? What mechanism increases or decreases the odds of damage to a Pi (not the SD, the Pi) if it is powered off before or after a Raspbian shutdown command?
Exactly. But I just chose that as an absolute worst case total loss situation on one side of the balance.
Huh again?
It seems we just agreed there is absolutely zero change in the odds of losing or not losing the hardware with shutdown or no shutdown. Therefore, hardware loss events do not enter into this discussion.
Yes? No?
All I was saying that even if the worst case was a total write off of your SD card and Pi it's ~50$ for the sake of argument. If something cost you 50$ and an hour of your time, how much effort would you be willing to put into mitigating it. It's just a thought experiment.
-
I guess in the context of using a Pi or similar as the SBC on a Duet3, it's pretty rare that anything is going to be written to the filesystem that's of any significance. Aside from doing a software update or uploading a gcode file, typical usage is pretty much just going to be read operations which aren't going to care about getting interrupted anyway.
The test is still really interesting to know how unlikely a failure is. I imagine that the more likely failure more will be due to SD dying due to swap spaces and temporary files being written... but that's a whole separate topic.While I'd always prefer to do a 'proper' shutdown, I've never felt the need on a Pi, and your test seems to have confirmed that. Though if there was an easy option to do so on the DWC, I'd likely use it.
-
@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:
Therefore, hardware loss events do not enter into this discussion.
Is this the can o worms phase of the discussion? I think I'll bow out now.
-
@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:
total write off of your SD card and Pi
I get the time 'cost' and completely agree with you. I don't get the "and Pi" part.
-
@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:
@gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:
Connect 12V to channel 1 and 5v to channel 2. Set the scope it to trigger on channel 1 falling edge.
Roughly what I was thinking. What voltage do we consider, as the 5V decays, to be the limit?
I've seen 4.8 and 4.75 as the lower limit. Does your scope allow a screen capture? See if you can just capture the slope.
-
The only time I get issues with corrupt sd cards on raspberry pi is when I attempt to upgrade RRF using sudo apt-get update, sudo apt-get upgrade.
(They not actually corrupt, it just doesnt end in success)Turning power off without shutting down pi causes less problems maybe.
-
@gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:
@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:
@gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:
Connect 12V to channel 1 and 5v to channel 2. Set the scope it to trigger on channel 1 falling edge.
Roughly what I was thinking. What voltage do we consider, as the 5V decays, to be the limit?
I've seen 4.8 and 4.75 as the lower limit. Does your scope allow a screen capture? See if you can just capture the slope.
Setting it up now.
-
Best I've gotten so far. Gnd to the ground to the common ground that everything shares, tip of 10x probe to pin 2 on expansion (5V). You can see the trigger setup in the upper right.
I can't go any slower on the horizontal without going into "slow acquisition mode", which I will try next.
Anything specific you were looking for?
Delta / Kossel printer fanatic
-
@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:
Modern file systems and flash storage devices just aren't as susceptible to power loss corruption as older filesystems on spinning magnetic storage.
That's certainly true of SSDs, which are designed to be tolerant of power failures. I'm not at all sure that it is true of SD cards, and they don't have the possibility of storing on-board power to enable them to shut down cleanly. So there may be a real danger that if you remove power while you are writing data to them, they could get corrupted. However, a Pi that is idle following a print probably isn't writing anything to the SD card; so that risk should be low. Also, the journaling file systems used by Linux are designed to be tolerant of unexpected removal of power.
-
Exactly, that's the reason Raspbian (and thus DuetPi) use ext4, which is a mature FS. Back in the days of ext1/ext2 power failures could provoke data loss but TBH I haven't managed to corrupt a single SD card or damage a Pi 3 or 4 just by plugging the power cord. When I did my initial tests with the Duet 3 attached, I even noticed that the Pi was staying on for a short moment after the power has been cut and sent a last message to DWC, so it may be even possible that the Linux kernel recognizes this and does a final flush before the power is completely gone.
-
Ext4 uses a mechanism called "write barriers" which ensures writing metadata correctly to the file system in case of power loss. Even for volatile write caches. This prevents data corruption.
-
@Danal said in Can-o-Worms: Pi Shutdown vs Power Off:
As of this writing, it has powered off more than 1460 times with no corruption to the SD card or file systems
It really depends on the use case, I'm using few opi's (same thing) and few rpi's with octoprint and the only time I had FS corruption (in years) was dead SD card. And they do write to log files and whatnot. On the other hand, I do some testing on 100+ of arm sbc's (opi's, rpi's, rock's...) and I do often power-cycle instead of a classic reboot (part of the test) and I get FS corruption in good 3-5% of cases (so approx more than once a week). So it really depends on what you do, how many files are open, do you have any fsync's running, how your FS is mounted (e.g. to get performance on database cluster you might use nobarrier) etc etc.. raspbian is as optimized as it gets to survive power-cycle and if you don't run weird stuff on it like I do in tests (database cluster
) the chance of FS getting corrupt is very very low.Now it can be done even better, if you look for e.g. how zeroshell works, when it boots up the only writable partition is in RAM disk, and if it needs to sync something to disk it remount RW, save, remount RO. In theory you could do this with rpi too but I doubt it is worth the trouble.
-
@Danal I was thinking more of the delay between 12v dropping and 5v dropping below the minimum for Pi operation. The trigger would be on the 12v channel and we'd see the slope on the 5v channel. Don't go to any more trouble though. It was curiosity on my part that I can easily satisfy by doing the same setup.
-
@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:
If anyone can show me a reliable way to KILL a pi or SD card (without actual negligence/malfeasance) I'd love to see it.
the marlin patch that write the position of the head to the sd card during print after each move and during long moves .. after a month printing SD card - dead, second month, second SD card dead, third month third SD card dead, then they called me to investigate what's killing their SD cards, I checked out what is "different" with the printer and found that option was on, turned it off, SD card never died again .. so there is your "relieable way to kill SD card"
-
@dc42 In this test, it was setup to write to a DB. I agree that an idle printer will write even less.
All that are suggesting read-only, or this or that: A core guiding principle of this test is to do nothing special to a Raspbian image on a Pi. Not even a configuration change. Whatever an end user is going to have by building a Pi per the Duet doc, that's what I wanted to test.
The only thing I "layered on" is the database stuff (borrowed from an existing monitoring system, and then timer-ized so there is a continuous feed of data), so that IO would unquestionably be higher than the printer use case.
As of this AM 2389 cycles and everything is still clean. The clicking of the relay is getting annoying, I'm stopping the test.
-
END OF TEST
At this point in time, I'm going to end the test. Of course, I don't "own" this discussion thread and anyone can continue as they like.
Total cycles: 2386+
Result: Clean file systems on SD card, clean database.My conclusions (YMMV):
Speaking only of a Raspberry Pi, running Raspbian, and the built-in SD card:
-
Best practice is to shutdown prior to power off. In those cases where you can't or don't shutdown, real-world risk of SD filesystem corruption seems very low.
-
SD cards fail for many reasons, including just wearing out. Backups, or other ways to recover, are essential. Shutdown vs. power down does not seem to significantly change this larger picture.
Thanks everyone for your discussion and ideas. This is a great group of people and discussions that would flame out in other places can occur here with only a spark or two, and that leads to productivity and learning.
-
-
@arhi said in Can-o-Worms: Pi Shutdown vs Power Off:
@Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:
If anyone can show me a reliable way to KILL a pi or SD card (without actual negligence/malfeasance) I'd love to see it.
the marlin patch that write the position of the head to the sd card during print after each move and during long moves .. after a month printing SD card - dead, second month, second SD card dead, third month third SD card dead, then they called me to investigate what's killing their SD cards, I checked out what is "different" with the printer and found that option was on, turned it off, SD card never died again .. so there is your "relieable way to kill SD card"
Death by a hundred thousand bytes.
-
I saw this yesterday: https://www.youtube.com/watch?v=IJn3fz7Zk4Q
It's a Pi4 case with passive/active cooling, and additional PCBs to redirect the ports all to one side. Plus it has a power button with software conditions, meaning you can do a long press and it will trigger a script to do a clean shutdown.
Even if you didn't want that particular case, it should be pretty easy to implement a physical button that could trigger a clean shutdown, which eliminates the hassle of remoting or using a display/mouse to shutdown.
