this post was submitted on 01 Aug 2024
2 points (100.0% liked)

Selfhosted

39980 readers
447 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

It's fairly obvious why stopping a service while backing it up makes sense. Imagine backing up Immich while it's running. You start the backup, db is backed up, now image assets are being copied. That could take an hour. While the assets are being backed up, a new image is uploaded. The live database knows about it but the one you've backed up doesn't. Then your backup process reaches the new image asset and it copies it. If you restore this backup, Immich will contain an asset that isn't known by the database. In order to avoid scenarios like this, you'd stop Immich while the backup is running.

Now consider a system that can do instant snapshots like ZFS or LVM. Immich is running, you stop it, take a snapshot, then restart it. Then you backup Immich from the snapshot while Immich is running. This should reduce the downtime needed to the time it takes to do the snapshot. The state of Immich data in the snapshot should be equivalent to backing up a stopped Immich instance.

Now consider a case like above without stopping Immich while taking the snapshot. In theory the data you're backing up should represent the complete state of Immich at a point in time eliminating the possibility of divergent data between databases and assets. It would however represent the state of a live Immich instance. E.g. lock files, etc. Wouldn't restoring from such a backup be equivalent to kill -9 or pulling the cable and restarting the service? If a service can recover from a cable pull, is it reasonable to consider it should recover from restoring from a snapshot taken while live? If so, is there much point to stopping services during snapshots?

top 4 comments
sorted by: hot top controversial new old
[–] [email protected] 1 points 3 months ago* (last edited 3 months ago)

I ran into a similar problem with snapshots of a forum and email server -- if there are scheduled emails when you take the snapshot they get sent out again if you create a new test server from the snapshot. And similarly for the forum.

I'm not sure what the solution is either. The emails are sent via an SMTP so it's not as simple as disabling email (ports, firewall, etc.) on the new test server.

[–] [email protected] 0 points 3 months ago

Check "green blue" deployment strategy. This is done by many businesses, where an interrupted service might mean losing a sale, or a client forever... I tried it sometime witj Nginx but it was more pain than gain (for my personal use)

[–] [email protected] 0 points 3 months ago (1 children)

If you're worried a out a database being corrupt, I'd recommend doing an actual backup dump of the database and not only backing up the raw disk files for it.

That should help provide some consistency. Of course it takes longer too if it's a big db

[–] [email protected] 0 points 3 months ago

I dump the db too.

With that said if backing up the raw files of a db while the service is stopped can produce a bad backup, I think we have bigger problems. That's because restoring the raw files and starting the service is functionally equivalent to just starting the service with its existing raw files. If that could cause a problem then the service can't be trusted to be stopped and restarted either. Am I wrong?