Shutdown printer and Raspberry Pi  

RE: Shutdown printer and Raspberry Pi

In theory, shutting down with the power switch can cause you to lose file system data and potentially require a fresh install onto the pi's SD card.

In practice, I've never had this happen to me, and I never manually power down my pi. 

RE: Shutdown printer and Raspberry Pi

Do you mean to switch off the printer via RPI or something else?

RE: Shutdown printer and Raspberry Pi

Hi, I have no straight solution for this. One thing that comes to mind is to try and re-use power panic to try and unmount the SD before power runs out, but i have no idea how successfull that might be...

RE:

There's too little time for that. You can however, if initiated from system do shutdown, and configure RPI to change state of some GPIO pin that would power off after shutdown.

Posted by: @tojik

Hi, I have no straight solution for this. One thing that comes to mind is to try and re-use power panic to try and unmount the SD before power runs out, but i have no idea how successfull that might be...

RE: Shutdown printer and Raspberry Pi

And how can I do that? The printer does not have the ability to turn itself off. You'd end up with a printer that is ON with rpi that is OFF. Furthermore, everyone would just ignore it, because that's too much hassle to do and they've got a switch anyway.

RE:

For example today I have the following setup:

- I have Gosund SP111 smart switch that is running esphome.io
- The printer is plugged into to SP111, SP111 into wall outlet
- The RPI is connected to printer 24V via buck converter
- Once the print finishes, the Klipper (I had the same on Octoprint before) does send a command to SP111 via HTTP API to turn off
- Since the SP111 is running via esphome a shutdown sequence starts on turn off
- SP111 will send a shutdown command to RPI via HTTP API (to Moonraker/Octoprint)
- The RPI executes `poweroff` to gracefully shutdown, it has 10s to do so (plenty of time)
- After 10s the SP111 will cut the power

To power on the printer:
- Press button on SP111 or go to SP111 web interface to turn on the printer + Raspberry PI

This as an added benefit provides wattage graph and very minimal consumption when not used. The Octoprint is best supported in general, as there are many plugins for power control, a lot provide a delayed shutdown sequence (as described above) that is supported by the most of power plugs (like TP-Link HS1xx).

Posted by: @tojik

The printer does not have the ability to turn itself off.

RE: Shutdown printer and Raspberry Pi

Nice setup you got going on there. But that's third party hw. I am pretty sure we're not going to dabble in that. You are free to script anything on the rpi tho.

I am approaching this from the stock setup perspective. I want to make that work as well as I can. We have some ideas to try in the safe shutdown department.

To answer the original question. For now, you'd have to make it shutdown using ssh before turning the printer off with a switch.

RE: Shutdown printer and Raspberry Pi

Actually to not depend on relay to allow RPI to safe-shutdown a few techniques might be used:

- mount root filesystem by default in read-only, so, the system will never get corrupted -> remount it to RW if you want to make changes
- have separate partition for print data (like gcodes) to be in sync or with very low cached write
- once you detect power loss do a sync remount into RO (`sysrq-trigger`) -> there might not be enough time for that

Now, detect power on failure. In worst case the data partition would be corrupted, but this is fsck fixable.  Nothing that cannot be recovered (especially if all data can be pulled from the internet and local storage is treated as a cache). The system will always be intact. Since the system would also be in read-only the chance of SD card failing is pretty much minimal/to non-existent. To perform updates you can remount in RW and do everything that is needed.

I do read-only with remount trick on a lot of my embedded systems where I don't trust the power delivery 🙂

Posted by: @tojik

Nice setup you got going on there. But that's third party hw. I am pretty sure we're not going to dabble in that. You are free to script anything on the rpi tho.

I am approaching this from the stock setup perspective. I want to make that work as well as I can. We have some ideas to try in the safe shutdown department.

To answer the original question. For now, you'd have to make it shutdown using ssh before turning the printer off with a switch.

RE: Shutdown printer and Raspberry Pi

Yup, that's been suggested internally as well. But that creates new problems. We might lose the rpi features like rpi imager settings. We would probably need to modify those scripts too (boot partition config ones). Sounds like a lot of work for not that much added functionality.

Thank you for sharing the details of an approach you have used successfully yourself. I've never done something like that, so having a sort of tutorial is awesome.

RE: Shutdown printer and Raspberry Pi

It's not the OS that loses the data, but the SD card controller.  The only reliable option is to use an industrial rated SD card, they don't perform dangerous operations in order to improve performance.  They are smaller and cost more but it doesn't matter as much for our application, we don't need 512G of storage after all.  I'm running on 8GB industrial cards and have yet to lose an image, I switch the machines on and off as I please.

RE:

Data can be lost everywhere: starting from CPU cache, in memory, or in SD controller.

Industrial cards have higher endurance mostly by using MLC or SLC instead of TLC for flash chips. However, this still does not prevent the loose of the data in a system that: uses journaling, uses writeback caches to improve write performance. For such, you should each time after unclean shutdown run fsck to ensure filesystem consistency.

Always, the clean shutdown should be done to ensure that all backlog is flushed to SD. Maybe in your case you not see a problem now, but someone due to non-zero probability will see it at some point. This might result in broken system, or crash during printing, like filesystem remounting due to unrecoverable error. Running fsck (often default) each time basically kills fast boot-up of the system, as fsck needs to be run before the FS is mounted.

The reason for doing read-only root is also to increase endurance to reduce amount of writes. This can be a good technique to allow basically any SD to be used or to reduce the likely of the catastrophic failure.

Posted by: @matthew

The only reliable option is to use an industrial rated SD card, they don't perform dangerous operations in order to improve performance. 

RE: Shutdown printer and Raspberry Pi

I think there is an error in your three letter abbreviations. I believe SLC is the most reliable, the more signal levels, the worse the longevity. MLC should mean multi level and TLC triple level.

But for the rest, I do know, I can lose data. Logs do not contain the minutes before the pi gets violently switched off. So either we are going to shrink the cache, sync after uploads and updates, or something. I still very much think it's important to do this without requiring the user to do anything extra and I went without losing anything back when I was using Octoprint, so I might be reluctant to focus all of my efforts on this specific thing, but some steps to prevent data loss should be made.

Doing a read only FS complicates everything else drastically, so I'd like to avoid that too if possible

RE: Shutdown printer and Raspberry Pi

Correct. SLC is a single bit, MLC is two bits per cell.

In general no need to do read-only as long as some things can be stored in memory, like logs. On some systems (OpenMediaVault) there's a a tool `folder2ram` which mounts all directories that can be ephemeral (/var/logs) using tmpfs to reduce amount of writes hitting SD.

Posted by: @tojik

I think there is an error in your three letter abbreviations. I believe SLC is the most reliable, the more signal levels, the worse the longevity. MLC should mean multi level and TLC triple level.

But for the rest, I do know, I can lose data. Logs do not contain the minutes before the pi gets violently switched off. So either we are going to shrink the cache, sync after uploads and updates, or something. I still very much think it's important to do this without requiring the user to do anything extra and I went without losing anything back when I was using Octoprint, so I might be reluctant to focus all of my efforts on this specific thing, but some steps to prevent data loss should be made.

Doing a read only FS complicates everything else drastically, so I'd like to avoid that too if possible

 

RE: Shutdown printer and Raspberry Pi

TL;DR: Should Prusa require an industrial SD card? no.  Should it be suggested?  I think so.  Should we do everything that we can to prevent data loss?  I believe that there are limits.

@k3213 yes, everything that you say is true and simply using an industrial SD card certainly won't save anything in the CPU cache, in memory or even in the SD controller.  The critical point that I'm trying to make is that a standard SD card can still corrupt your block device during a brown-out event.  I am well aware of what can happen when a disk fails to write data properly.  I ran the tests for brad@livejournal when the drives were lying to us about writes.  No amount of journalling will save you here, you will eventually have an unrecoverable error.  If an MLC block is cleared and not rewritten during a power event then you just lost data that was already on disk.  For a video camera this generally doesn't matter, not the same story for a Linux system.

Industrial cards use SLC, have wear-levelling and are generally hardened for power loss, extended write cycles and whatnot.  WD has a document talking up their cards.

@Tojik I agree that we want to keep all the things you listed.  I'd also be wary of trying to modify the OS to support a more industrial installation.  Obviously we should do as much as we can to prevent data loss, but I'd personally rather have you improving PrusaLink than mucking with the base OS.  A second partition for data *might* help?  Of course if your superblock is in the wrong MLC cell then you're still going to have a bad day.

RE: Shutdown printer and Raspberry Pi

Hello,

Bigtreetech has a Power Panic like DC-DC converter for RPI.
https://github.com/bigtreetech/SKSM

It's around 20 Euro's on Aliexpress

RE:

Good discussion so far!

Given that a Prusa 3D printer is a consumer machine, it is assumed that users will treat it as such and turn it on and off as required. The classic worst case scenario is as follows:

I make changes to a series of parts that need to be reprinted and upload them all to PrusaLink internal storage intending to start printing in the morning: I then turn off the printer and go to bed without realising that I have a non-consumer computer built into the printer. In this scenario, with a large volume of writes to the rPi SD card, there is a pretty good chance I will lose some of the files written at the least and corrupt the SD card at the worst.

Now, since I know enough about Linux/Raspberry to know better, I connect to the rPi via putty and shutdown Linux before turning off my printer.

I would imagine that most people who undertake the rPi upgrade to their MK3S(+) as I have, would understand the necessity to treat the rPi as a computer, rather than a coffee machine (hmm, I wonder if I can upgrade it to make coffee??).

I would also imagine that many, many users of Prusa machines, have no idea of the necessity to properly eject SD cards from computers and consequently, end up with corruptions and short gcode files on their SD cards.

Regardless, I believe it is in Prusa's interests to do all they can to protect consumers from themselves with PrusaLink in particular, as it will save a lot of headaches in the future. 

Part of the problem is that there are multiple locations for gcodes to reside, which in itself is a problem because it requires more rigour from the user to ensure there are no costly mistakes made by printing a file unintentionally. I believe the whole file handling aspect of Connect/PrusaLink/SD Card/Local Storage needs to be addressed. I can see why PrusaLink storage on a per-printer basis is needed in a farm situation, but it really is superfluous in a single-printer environment. Perhaps PrusaLink storage could optionally be held in a RAM-based temporary file system. This satisfies the need for the file to be transferred to PrusaLink for printing and protects the OS from crash shutdowns. 

RE:

I prefer using proper ups hat on the pi with lipo batteries instead of capacitors - mainly because some things that I do on pi take a bit of time and thus more power to be aborted and shut down safely.