Duet3D Logo

    Duet3D

    • Register
    • Login
    • Search
    • Categories
    • Tags
    • Documentation
    • Order

    Can-o-Worms: Pi Shutdown vs Power Off

    Duet Hardware and wiring
    15
    58
    640
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Danal
      Danal last edited by Danal

      SUMMARY

      I built a 'test rig' to attempt to quantify the operational risk of powering off a Pi without a shutdown command. It uses an Arduino pro mini and a relay to control power to a Raspberry Pi 3B+. It does do IO to the SD on a regular basis (see details next post).

      As of this writing, it has powered off more than 1460 times with no corruption to the SD card or file systems. It is still cycling; I am posting because 1460 is equivalent to 4 times per day for one year.

      MOTIVATION

      I am not 100% clear in my motivation for doing this, and even less clear about posting the results on the Duet forum. It is clear that:

      Some people, including me, are going to power cycle Pis. Other people are going to shutdown first.

      I do not wish to "force" change on anyone; at the same time, I'd like to see some facts on the table for those who are still making their choice. In analyzing my motivations, I anticipate that at least some people will see the test as invalid. And the discussion around the test setup may become deeper than the shutdown/poweroff discussion itself. I believe the test setup is valid... see details next post... and I am very willing to discuss more detail or improvements or etc.

      CONCLUSION

      For me, the operational risk of powering of a Pi without a shutdown is acceptable.

      I fully acknowledge that there is no "proof" here and that the SD could corrupt on the next power cycle. Always backup.

      Delta / Kossel printer fanatic

      deckingman arhi 2 Replies Last reply Reply Quote 1
      • Danal
        Danal last edited by Danal

        TEST SETUP

        The Pi:

        • Raspberry Pi 3B+ About two years old.
        • SD card is Sandisk "edge" 8GB. Not sure where it came from. Nothing special.
        • OS is imaged from the RaspberryPi foundation web site, regular Raspbian
        • Standard install for ssh, wireless, etc.
        • Nothing special done to file systems or checking similar.
        • Also installed 'sqllite3' DB and a bunch of scripts that I use on a different Pi as a monitor.
        • Changed the scripts to generate fake data every 2 seconds
        • This will result in plenty of activity to the SD
        • This actual monitor generates transactions every 15 seconds to 5 minutes
        • Therfore, the 'fake' data writes to the SD 30x to several hundred x more than the real monitor.

        There is also a script that scans the boot log, and finds the entry 'systemd-fsckd.service: Succeeded.' (or failed). This is the file system check that is built into Raspbian. No change to the default image. Once this entry is found, it is pushed to a CIFS (windows) network shared drive and appended to a log file. That is my main way of knowing how many boot cycles have occured. It also validates that MANY things on the PI/Card are working, not corrupted.

        The power cycle:

        • Eventek KPS3010D benchtop power supply set to 12v.
        • Arduino micro pro running small custom sketch.
        • Sketch closes relay (power on) for two minutes, opens relay (power off) for 30 seconds, repeats.
        • Relays are ?songle? 10A at 50V DC
        • There are two relays so that + and - power wires are both cut.
        • The relays are driven from Arduino IO 3 and 4, via a FET
        • 12V power goes from the relay to a "UBEC" that outputs 5.1V up to 3A
        • Power wires from the UBEC are soldered to the Pi on the back of the board, near the micro USB power connector (reused a prior setup)
        • There is also a button that signals the Arduino script to not cycle off, so that I can ssh into the Pi and fix anything, if required (so far, not)

        This was re-used from a setup from years back, so it has extra stuff not relevant to this test; it was just a handy way to get it set up quickly:
        PiPowerTestRig2.jpg

        Delta / Kossel printer fanatic

        1 Reply Last reply Reply Quote 0
        • Danal
          Danal last edited by Danal

          Other thoughts on the test rig:

          • As stated over and over, this is "nothing special", every effort to make it a regular Pi Raspbian build.
          • It DOES do IO to the card. In fact, averages a lot more than the Pi in an idle printer will ever be writing.
          • Some people may ask for even more IO. This would be unrealistic. Very heavy IO shouldn't be done to an SD card; if your Pi is going to be doing tons of IIOPS, boot from an SD and read/write to a USB disk of some sort.

          What the test is not:

          • It is NOT "proof". Neither in the formal or informal sense.
            • For one thing, there is no control group
            • For another, you cannot 'prove' a negative. Logic 101.
          • It is NOT designed to separate out events caused by the fact that SD cards WILL eventually fail from write activity
            • To be clear, no such events have occurred
          • It is NOT based on any bench-marking tools. For example, the things I picked to do IO are completely arbitrary
          • It is NOT "deep scanning" the file systems.
            • The definition of 'success' is 'the boot fsck is happy' and that the Pi 'operates normally over time'.
            • That second one is the real key.
            • If corruption occurs and is not fixed, it should get worse very quickly, resulting in an un-runnable system.
          • It is NOT certified or audited by anyone at all. You have to trust me that I'm not lying or witholding any part of the data.
          • It is NOT the way you would do it.
          • It is NOT intended to eliminate all risk of losing the SD contents. Always backup.

          What the test is:

          • It is statistical in nature.
          • It is intended to be as realistic as possible.
          • It is intended to run enough times to quantify operational risk.
          • It is sitting there continuing to yank power every 2.5 minutes.

          I don't know if I will run it to failure, or not. It is taking up a bit of bench space and a power supply.

          Delta / Kossel printer fanatic

          1 Reply Last reply Reply Quote 0
          • Luke'sLaboratory
            Luke'sLaboratory last edited by

            If you do come up with a result, I'd be curious to see what the MTTF is for a similar board, the Odrid C2, which can upgrade to using EMMC modules instead of sd cards.

            Luke
            http://lukeslab.online

            A Former User 1 Reply Last reply Reply Quote 1
            • chas2706
              chas2706 last edited by

              @Danal

              Just for your info,

              until recently upgrading one of my printers to Duet 3 + SBC I had a raspberry pi3 + usb camera on the old set up running motioneyeos.

              Because the pi's are so cheap I never bothered to shutdown the pi in the correct way.

              When i'd finished printing I would just power the printer off which would immediately cut power to the pi. (I fed it from a 5v buck converter hooked up to my printer psu).

              I used this set up for approx. 2 - 3 years and I can confirm that I never had any issues.
              The sd card has never corrupted.

              In fact it is the same rpi3, sd card and motioneyos.img that is now running on my new duet 3 set up. (I have rpi4 as the SBC).

              Luke'sLaboratory 1 Reply Last reply Reply Quote 0
              • Luke'sLaboratory
                Luke'sLaboratory @chas2706 last edited by

                @chas2706 brings up a good point. I don't think I've ever properly shut down my pi on any of my projects (reboots notwithstanding) that goes back 5+ years. All of my octoprint installs are all on the same power supply as my printers, and they'll hit probably a cycle a week or so.

                My duet3-based system however gets power cycled on average twice a day.

                Luke
                http://lukeslab.online

                1 Reply Last reply Reply Quote 0
                • gtj0
                  gtj0 last edited by

                  @Danal Out of curiosity, how much time elapses between the 12v cutout and the 5v dropping below the Pi's Vin minimum?

                  Danal 1 Reply Last reply Reply Quote 0
                  • chas2706
                    chas2706 last edited by

                    @Luke-sLaboratory said in Can-o-Worms: Pi Shutdown vs Power Off:

                    brings up a good point. I don't think I've ever properly shut down my pi on any of my projects (reboots notwithstanding) that goes back 5+ years. All of my octoprint installs are all on the same power supply as my printers, and they'll hit probably a cycle a week or so.
                    My duet3-based system however gets power cycled on average twice a day.

                    Yes I can well believe it.

                    I have played around with raspberry pi's since they first came on the market.
                    Done loads of projects with them and used many different versions of os's and cannot recall anytime where the sd card(s) became corrupt upon power failure.

                    I also have a home made raspberry pi cctv system that runs 24/7 and has done for the last 6 years and has experienced some power failures but without issue.

                    1 Reply Last reply Reply Quote 0
                    • deckingman
                      deckingman @Danal last edited by

                      @Danal Please excuse my ignorance because my RPi although fitted, remains unconnected so I have zero experience of using the thing. I assume that the reason why you have gone to these lengths is because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" yes?

                      Ian
                      https://somei3deas.wordpress.com/
                      https://www.youtube.com/@deckingman

                      DanS79 Danal 2 Replies Last reply Reply Quote 0
                      • chas2706
                        chas2706 last edited by

                        @gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:

                        Out of curiosity, how much time elapses between the 12v cutout and the 5v dropping below the Pi's Vin minimum?

                        On my new set up I have a separate beefy 5v 10 amp psu to supply my 2 pi's .
                        They both now do a proper shutdown. I purchased a 5v relay module of which the contacts are connected to the 230v live to the system.
                        I have a rpi 4 as the SBC and a rpi 3 running motioneyeos for my webcam.

                        The relay input is connected to a spare gpio pin (pin18) on the SBC pi ( I got rid of the ribbon connector that stops you using the unused gpio pins and made my own 6 wire connector).
                        The relay board is fed from 3.3v on the pi and is active low.

                        On the SBC pi I set pin 18 as active low on start up and made a small script called "shutdown pi's" which is accessible from the pi start menu.
                        The script firstly connects to the rpi 3 via ssh and gives it the shutdown command then there is a deliberate time delay before the rpi 4 is given the shutdown command.

                        It all works a treat and I think it is down to having the rpi's fed from external (beefy) psu because both rpi's have more than enough time to shut down completely before the external psu loses all of its power!

                        1 Reply Last reply Reply Quote 0
                        • DanS79
                          DanS79 @deckingman last edited by

                          @deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

                          Please excuse my ignorance because my RPi although fitted, remains unconnected so I have zero experience of using the thing. I assume that the reason why you have gone to these lengths is because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" yes?

                          A Pi really isn't really any different than your desktop. If you suddenly loose power, you can corrupt the file system under the right conditions. Depending on how your system is configured and what its doing at the time of power loss, your odds of corruption could be anywhere from guaranteed to 1 in a trillion.

                          It really depends on how much you care about potentially loosing data.

                          For example if not loosing data is important to you, you could do something like this.
                          https://www.youtube.com/watch?v=m20o3P-S5s8

                          deckingman 1 Reply Last reply Reply Quote 0
                          • A Former User
                            A Former User @Luke'sLaboratory last edited by

                            Admittedly never had a Pi corrupt a card, but plenty of other linux based systems packing in due to power loss, so I'm not about to call it a theoretical problem or stop shutting down Pi's cleanly, if I couldn't be bothered to run poweroff I'd just leave the thing running.

                            If the Pi and accompanying software were sufficiently stable you could simply run sudo raspi-config and enable read-only filesystem. But with new updates weekly or every other day that becomes cumbersome if not integrated in the upgrade scripts.

                            In general the whole Pi setup needs hardening and I'm sure it will come, but I also suspect the team needs to make it usable first.

                            Re: EMMC; ask Tesla what they think; although their issues are more related to write (erase) operations than shutdowns.

                            1 Reply Last reply Reply Quote 0
                            • Phaedrux
                              Phaedrux Moderator last edited by

                              I see your reckless Pi shutdowns and raise you about 50 Atom CPU based windows 10 mini PCs that get unceremoniously power killed on a regular basis (power outages, regular maintenance, etc) for the past 3 years and have not had any OS corruption at all. They use an onboard eMMC flash storage device (basically an SD card soldered to the mobo). Modern file systems and flash storage devices just aren't as susceptible to power loss corruption as older filesystems on spinning magnetic storage.

                              Z-Bot CoreXY Build | Thingiverse Profile

                              garyd9 dc42 2 Replies Last reply Reply Quote 3
                              • deckingman
                                deckingman @DanS79 last edited by

                                @DanS79 Yes, I get that. But it doesn't answer my question.

                                @Danal has gone to the length of building a test rig and after much evaluation has arrived at the conclusion (for him personally) quote "For me, the operational risk of powering of a Pi without a shutdown is acceptable.".

                                Now @Danal doesn't strike me as being the sort of person who builds test rigs unless there is some reason for doing so. In which case, I concluded that there must be a reason why he prefers to "pull the plug" rather than doing a controlled shutdown. So my question remains. - "Is it because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" ?

                                Ian
                                https://somei3deas.wordpress.com/
                                https://www.youtube.com/@deckingman

                                Phaedrux A Former User 2 Replies Last reply Reply Quote 0
                                • Danal
                                  Danal @gtj0 last edited by

                                  @gtj0 said in Can-o-Worms: Pi Shutdown vs Power Off:

                                  @Danal Out of curiosity, how much time elapses between the 12v cutout and the 5v dropping below the Pi's Vin minimum?

                                  I will see if there is a reasonable way to measure that. For example, if my dual channel scope will go slowly enough to trigger on the 12V and also draw a trace for the 5V.

                                  If you have any thoughts on how to measure...?

                                  Delta / Kossel printer fanatic

                                  gtj0 1 Reply Last reply Reply Quote 0
                                  • Phaedrux
                                    Phaedrux Moderator @deckingman last edited by Phaedrux

                                    @deckingman I think the downsides come down to complexity and time. Since the pi isn't always using a display and keyboard and mouse it can take time to get it shut down cleanly because you might have to remote into it on your phone or other PC or connect a display and mouse. Plus it also means waiting for the system to shut down. Or you could maybe wire up some sort of trigger relay to shut it down cleanly with a button press, but you still have to wait for that.

                                    BUT if there is really negligible risk to simply killing the power, you can just unplug it or flip a power switch, or have it wired alongside the Duet which gets killed when power is removed to the printer, and not really worry about it.

                                    Z-Bot CoreXY Build | Thingiverse Profile

                                    Danal A Former User 2 Replies Last reply Reply Quote 0
                                    • A Former User
                                      A Former User @deckingman last edited by

                                      @deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

                                      So my question remains. - "Is it because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" ?

                                      no, there are many ways to achieve controlled shutdown - but there are tradeoffs with respect to security and at the end of the day it doesn't protect against power loss (unless you add battery backup to the mix)

                                      1 Reply Last reply Reply Quote 0
                                      • garyd9
                                        garyd9 @Phaedrux last edited by

                                        @Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

                                        Modern file systems and flash storage devices just aren't as susceptible to power loss corruption as older filesystems on spinning magnetic storage.

                                        While I tend to agree to some extent, any reasonable general purpose caching filesystem can be corrupted from a power loss, regardless of the backing storage technology. If power is lost in the middle of a write, corruption can occur. A filesystem check on a filesystem with a journal often will "repair" the filesystem, but without knowing the intent of the programs writing data, it might be incapable of properly repairing the data. (A classic example is a database with bad indexes due to a power loss between data and index writes.)

                                        I'll point out the duet wiki page of How to Destroy your Duet 2. In that document, there's a clear disclaimer of "We can't guarantee that these techniques will not destroy your Duet 2" and " doing anything described here will invalidate your warranty!"

                                        The same goes for unplugging your raspberryPi without shutting down the filesystems cleanly. Just because it "probably" won't cause you headaches, or just because it didn't cause problems X times before, doesn't mean it's safe or a good idea.

                                        "I'm not saying that you are wrong - I'm just trying to fit it into my real world simulated experience."

                                        Phaedrux 1 Reply Last reply Reply Quote 0
                                        • Danal
                                          Danal @deckingman last edited by Danal

                                          @deckingman said in Can-o-Worms: Pi Shutdown vs Power Off:

                                          @Danal Please excuse my ignorance because my RPi although fitted, remains unconnected so I have zero experience of using the thing. I assume that the reason why you have gone to these lengths is because a controlled shutdown is difficult to achieve, or has some other downsides compared with simply "pulling the plug" yes?

                                          Two very separate reasons I built the test rig.

                                          1. When this gets discussed (here and other forums), and I say 'I just power off', some people react as though my Pi is in control of a nuclear bomber overflying their neighborhood. Seriously, you'd be amazed at how nasty some people have been that I might be misleading noobys or something. This forum is pretty respectful, many are not.

                                          2. Reason two is really a whole laundry list:

                                          • I'm just lazy.
                                          • I have more Pi's than just the one in the printer. Some of them are easy to access for a "sudo shutdown". Many of them are not.
                                          • I'm just lazy.
                                          • The need for shutdown of journaling file systems has struck me as BS for decades. I recognize that some system really do need to on a "pragmatic" basis (high risk of corruption)... I've always believed philosophically that was unacceptable design on the part of the OS and Filesystem designers. Foisting a file system on the public that is beyond its own provided "check at power up" is almost criminally negligent and is absolutely technically arrogant.
                                          • I'm just lazy.
                                          • And I could have been wrong. This test rig could have demonstrated that corruption is 1:10 or 1:100 odds, not the 1:thousands that it seems to show.

                                          And mostly, I'm just lazy.

                                          Delta / Kossel printer fanatic

                                          deckingman 1 Reply Last reply Reply Quote 0
                                          • Danal
                                            Danal @Phaedrux last edited by

                                            @Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

                                            BUT if there is really negligible risk to simply killing the power, you can just unplug it or flip a power switch, or have it wired alongside the Duet which gets killed when power is removed to the printer, and not really worry about it.

                                            Exactly.

                                            Delta / Kossel printer fanatic

                                            1 Reply Last reply Reply Quote 0
                                            • A Former User
                                              A Former User @Phaedrux last edited by

                                              @Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

                                              and not really worry about it.

                                              read only file system gives peace of mind, statistics not so much

                                              1 Reply Last reply Reply Quote 0
                                              • Phaedrux
                                                Phaedrux Moderator @garyd9 last edited by

                                                @garyd9 said in Can-o-Worms: Pi Shutdown vs Power Off:

                                                The same goes for unplugging your raspberryPi without shutting down the filesystems cleanly. Just because it "probably" won't cause you headaches, or just because it didn't cause problems X times before, doesn't mean it's safe or a good idea.

                                                Totally agree, and that's where the risk must be weighed. How much time will be wasted in the case of a failure? How can that time be mitigated with backups, etc?

                                                This is where I have a hard time because I haven't yet been able to cause the failure situation in normal usage, so I'm not really sure how much pain would go into a recovery. Worst case scenario for a Pi would be maybe replacing the SD card AND the pi itself. So that's maybe 50$ financial hit, plus the time to download an up to date pi image and install it on the fresh SD card, update the system, restore any backups or do a fresh install of whatever software.

                                                In the case of the 50 mini pcs I mentioned, the risk was deemed acceptable, because the cost and complexity of mitigation was far too high to justify. If a mini PC dies it simply gets replaced with a spare. At 200$ each it's far more than a pi, but it's so rare, and the mitigation impossible (battery backup is out of the question in this locale), that it's an accepted risk.

                                                If anyone can show me a reliable way to KILL a pi or SD card (without actual negligence/malfeasance) I'd love to see it.

                                                Z-Bot CoreXY Build | Thingiverse Profile

                                                garyd9 A Former User Danal arhi 5 Replies Last reply Reply Quote 0
                                                • garyd9
                                                  garyd9 last edited by garyd9

                                                  Not sure what kind of writes your doing in your test, but perhaps try something a bit more annoying to filesystems: zip. (might need to "apt install zip")

                                                  The reason is that the headers in a zip archive have to be updated after compressed data is written to the file, so there's a chance that the archive header is out of date with respect to the data in the archive, rendering the entire archive corrupted (even if the filesystem isn't corrupted.)

                                                  Just run "zip test.zip /usr/bin/*" and on OS startup, test the zip for integrity.

                                                  "I'm not saying that you are wrong - I'm just trying to fit it into my real world simulated experience."

                                                  1 Reply Last reply Reply Quote 0
                                                  • garyd9
                                                    garyd9 @Phaedrux last edited by

                                                    @Phaedrux said in Can-o-Worms: Pi Shutdown vs Power Off:

                                                    Totally agree, and that's where the risk must be weighed. How much time will be wasted in the case of a failure? How can that time be mitigated with backups, etc?

                                                    I'm not familiar with what DSF/DSC is doing behind the scenes, but a question is what it might be writing to the filesystem. Anything in /var/log can be redirected to a tempfs (it'd be lost on power failure.) If that's all the fs writing that that the duet stuff does (other than writing gcode files in direct response to user interaction), then it should be reasonably safe. I'd probably turn off write caching on the file system as well (with nearly zero penalty if logs are all going to ramdisk.)

                                                    "I'm not saying that you are wrong - I'm just trying to fit it into my real world simulated experience."

                                                    1 Reply Last reply Reply Quote 0
                                                    • deckingman
                                                      deckingman @Danal last edited by

                                                      @Danal said in Can-o-Worms: Pi Shutdown vs Power Off:

                                                      ....................Seriously, you'd be amazed at how nasty some people have been that I might be misleading noobys or something. This forum is pretty respectful, many are not.

                                                      Ohh, I'm not amazed - I have my own blog and YouTube Channel so I know how true that is.

                                                      And mostly, I'm just lazy.

                                                      Thanks for your honesty. I'm even more lazy - maybe that's why I'm sticking to stand alone "RPi-less" operation. If I don't start it up, I don't have to worry about shutting it down ☺

                                                      Ian
                                                      https://somei3deas.wordpress.com/
                                                      https://www.youtube.com/@deckingman

                                                      1 Reply Last reply Reply Quote 1
                                                      • First post
                                                        Last post
                                                      Unless otherwise noted, all forum content is licensed under CC-BY-SA