this post was submitted on 08 Feb 2025
125 points (97.7% liked)

Selfhosted

42535 readers
644 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

I have backups on a backup hard drive and also synced to B2, but I am thinking about backing up to some format to put in the cupboard.

The issue I see is that if I don't have a catastrophic failure and instead just accidentally delete some files one day while organising and don't realise, at some point the oldest backup state is removed and the files are gone.

The other thing is if I get hit by a bus and no one can work out how to decrypt a backup or whatever.

So I'm thinking of a plain old unencrypted copy of photos etc that anyone could find and use. Bonus points if I can just do a new CD or whatever each year with additions.

I have about 700GB of photos and videos which is the main content I'm concerned about. Do people use DVDs for this or is there something bigger? I am adding 60GB or more each year, would be nice to do one annual addition or something like that.

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 20 points 1 week ago* (last edited 1 week ago) (2 children)

I'm using blu-ray disks for the 3rd copy, but I'm not backing up nearly as much data as you are.

The only problem with optical media is that you should only expect it to be readable for a couple of years, best case, at this point and probably not even that as the tier 1 guys all stop making it and you're left with the dregs.

You almost certainly want some sort of tape option, assuming you want long retention periods and are only likely to add incremental changes to a large dataset.

Edit: I know there's longer-life archival optical media, but for what that costs, uh, you want tape if at all possible.

[–] [email protected] 10 points 1 week ago

Same.

Bought a Blu-ray burner and "archive grade" disks for third location backups.

I made a list of files that is just a text document (3MB!) that sits on the root of the Blu-ray. There's probably a better way of doing that, but it works for me.

[–] [email protected] 3 points 1 week ago (13 children)

Hmm I am keen for something that could be left in the cupboard for 50 years and still works when brought out.

What does it take me to do home tape storage? Do the tapes needs to be stored with climate control or are they pretty stable? Is it feasible for the average person to load the contents?

I'm thinking of pulling a suitcase out of the cupboard of all the baby photos, but digital files or photo and video.

[–] [email protected] 7 points 1 week ago (1 children)

So, 50 years isn't a reasonable goal unless you have a pretty big budget for this. Essentially no media is likely to survive that long and be readable unless they're stored in a vault, under perfect climate controlled conditions. And even if the media is fine, finding an ancient drive to read a format that no longer exists is not a guaranteed proposition.

You frankly should be expecting to have to replace everything every couple of years, and maybe more often if your routine tests of the media show it's started rotting.

Long term archival storage really isn't just a dump it to some media and lock it up and never look at ever again.

Alternately, you could just make someone else pay for all of this, and shove all of this to something like Glacier and make the media Amazon's problem. (Assuming Amazon is around that long and that nothing catches fire.)

[–] [email protected] 2 points 1 week ago (14 children)

Hmm damn. I don't really think cloud is the right answer for what I'm trying to do.

I disagree that formats like JPEG won't be readable in 50 years. I feel like there would be big demand for being able to read the format even if it's been superceded, on account of all the JPEGs that still living people have.

Maybe I get a big drive. Each year I copy over files from the last year. Every X years I swap the hard drive for a new one, copy all data.

How can I tell if individual files get corrupted? Like the hard drive failed in that section, then I copy the corrupted file to the new drive, and I'd never know. Can I test in bulk? 50k+ photos and videos so far.

[–] [email protected] 7 points 1 week ago (3 children)

The format is the tape in the drive, or the disk or whatever.

Tape existed 50 years ago: nothing modern and in production can read those tapes.

The problem is, given a big enough time window, the literal drives to read it will simply no longer exist, and you won't be able to access even non-rotted media because of that.

As for data integrity, there's a lot of options: you can make a md5 sum of each file, and then do it again and see if anything is different.

The only caveat here is you have to make sure whatever you're using to make the checksums gets stored somewhere that's not JUST on the drive because if the drive DOES corrupt itself, and your only record of the "good" hashes is on the drive, well, you can't necessarily trust those hashes either.

load more comments (3 replies)
load more comments (13 replies)
load more comments (12 replies)
[–] [email protected] 13 points 1 week ago (8 children)

I have not used them myself, but M-DISC sounds like what you’re looking for. There are a few other alternatives listed on that Wikipedia article, too.

load more comments (8 replies)
[–] [email protected] 9 points 1 week ago* (last edited 1 week ago) (1 children)

I usually use a dehydrator for ~3 days on my drives to make them shelf stable. So far I haven’t had any issues.

[–] [email protected] 2 points 1 week ago

This will do nothing at all. Drives don't die by rust. They usually die because the motor somehow can't get the discs to spin. Very often dry lube is the reason. That can occur if you leave the drive off too long.

[–] [email protected] 8 points 1 week ago* (last edited 1 week ago) (2 children)

Tape. Amazon glacier if you're okay with that.

And regular test restores. An untested backup is not a backup.

But when considering what I need to back up, I usually overestimate how much I or other people will care if it's lost. Family photos are great, but what are the odds of someone saying "damn I wish we still had two dozen photos of that one barbecue?"

[–] [email protected] 5 points 1 week ago (2 children)

I use tape but haven't been happy with my drive for a while, where do you get your drives? (Also OP I wouldn't recommend tape until you cross the 10TB mark personally)

[–] [email protected] 4 points 1 week ago

Yeah after looking at the price of a drive, I agree it doesn't seem necessary at the level of data I have.

load more comments (1 replies)
load more comments (1 replies)
[–] [email protected] 7 points 1 week ago* (last edited 1 week ago) (1 children)

Currently the only solution for a consumer are M-Disc Blue rays. They are currently the only "write once read many" media available that are preferable in these types of situations.

The media is comparable cheap - you can safe your amount of data for around 80-90USD/€ initially(or less for more but smaller discs) and then pay around 10$/€ per year for the new amount of data.

The chances that in 20 years someone is still able to read them are fairly high - there are numerous businesses that are using these disc as WORM media to backup important data on a medium that a opposing lawyer later cannot claim "was manipulated". In 50 years it is very likely to be readable at least by professionals. The discs itself are rated for much longer storage.

If you write on them unencrypted there should be no problem of writing on them. Additionally they do not have issues with byte rot,etc.

[–] [email protected] 3 points 1 week ago (1 children)

Yeah it's an interesting thought. They seem to come up to 100GB capacity, but the wikipedia page claims (with a [dubious] qualifier) that you need some sort of special higher power burning device to write to M-Disc.

I don't have an optical drive at the moment. Would I just pick any rated for BDXL?

[–] [email protected] 3 points 1 week ago

You need a designated M Disc capable burner,yes. (Not generic BDXL,there are slight differences) There are a few on the market though - they cost around 100-150 bucks usually.(In theory you can use a regular writer sometimes - I know people who do that,but why risk that?) I usually recommend the verbatim to my clients,they are dirt cheap and work flawlessly so far.

For reading the discs any regular data-capabale blue ray disk drive will do.

[–] [email protected] 6 points 1 week ago* (last edited 1 week ago) (3 children)

The other thing is if I get hit by a bus and no one can work out how to decrypt a backup or whatever.

Documentation, documentation, documentation. No matter what system you have, make sure your loved ones have a detailed, image-heavy, easy to follow guide on how restorations work - at the file level, at the VM level, at whatever level you are using.

That being said, DVDs actually have quite a short shelf life, all things considered. I'd be more inclined to use a pair of archival strength USB NVME drive, updated and tested routinely(quarterly, yearly, whatever makes sense). Or even an LTO tape, if you want to purchase the drive and some tapes.

You can put your backups in something like VeraCrypt. Set an insanely long password, encoded in a QR code, printed on paper. Store it in the same secured location you store your USB drives (or elsewhere, if you have a security posture).

You may also consider, if money is not a concern, a cloud VPS or other online file storage, similarly encrypted. This can provide an easy URL to access for the less tech-savvy, along with secured credentials for recovery efforts. Depending on what your successors might need to access, this could be a very straightforward way to log into a website and download what they need in an emergency.

[–] [email protected] 4 points 1 week ago (1 children)

As much as I'm worried about family not being able to do it, I'm just as worried that I will do something dumb and lose the encryption key, losing everything. I am keen on the digital equivalent of a suitcase full of photos that could be stumbled upon.

I also already have borg backup set up to a backup drive and synced to the cloud (Backblaze B2).

For tape drives, is many thousands of dollars a normal price? Not sure I'm that keen.

[–] [email protected] 3 points 1 week ago (1 children)

If you buy your LTO drive new, then yes they rip you a new one, for sure! Buy it used...but it still will cost you a few hundred. Like I said, if money is not a concern. If losing the encryption key is a concern, then USB is still your best bet. Make two, keep them simple and unencrypted, stick em in two different safes, update them regularly. And print the documentation with pictures!

load more comments (1 replies)
load more comments (2 replies)
[–] [email protected] 6 points 1 week ago (3 children)

This was a recent point of discussion on the 2.5 Admins podcast (https://2.5admins.com/2-5-admins-228/). Some good discussion on there.

My own thought is the best way to handle your family-member-finding-your-old-photos problem is the analog way: make some prints. It’s absolutely idiot proof, the methodology of keeping paper goods is well understood, and the technology is platform independent.

load more comments (3 replies)
[–] [email protected] 6 points 1 week ago (3 children)

After reading the previous discussion I think that you should get more than single drive to store cold backups. That way you can at least spread out the risk of single drive failing. 2TB spinning drives are pretty cheap today and if you have, for example, 4 of them, you can buy one now, write your backups to it and in 6 months buy another, write data on that and so on.

This way you'll have drives with year or two difference on purchase date, so it's pretty unlikely all of them fail at once and a single drive gets powered on and checked every other year or so. My personal experience is that spinning drives are pretty stable on the shelf, but I wouldn't rely on them for decades. And of course even with multiple drives you'll still want to replace them every 3-5 years each. Plus with multiple drives, if I were to build setup like that, I'd set up some sort of scripts or other solution where I can just plug the thing in and doubleclick an icon on desktop to refresh the data and maybe get a notification automatically that the drive you're using should be replaced.

And for actual, long term storage, printouts are the way to go. At least in here you can get books made out of photo paper with your pictures. That's one media which is actually stable over long period and using them doesn't require a lot of technical knowledge nor hardware. But I'd still keep digital copies around, as the printouts aren't resistant to things like house fire or water damage.

load more comments (3 replies)
[–] [email protected] 6 points 1 week ago (1 children)

For your amount, just an external hard drive attached to a NAS or something is fine, or a 2-bay synology would be more than enough. Drives are coming in 20-24TB models now, that'd keep you going for a long time.

load more comments (1 replies)
[–] [email protected] 5 points 1 week ago (1 children)

USB hard drive? If we're talking about a cold backup that's easy to access a USB drive is reliable and easy.

load more comments (1 replies)
[–] [email protected] 5 points 1 week ago* (last edited 1 week ago) (1 children)

In your scenario, I'd be looking at ZFS or BTRFS for your live data, especially when taking photos into account. They'll self-repair files that may run into decay issues, which I've seen a lot of with photos in all formats. Since you already keep off-site backups, I'd then just keep an extra drive around that you snapshot to from time to time.

[–] [email protected] 2 points 1 week ago (1 children)

So my offsites are an incremental backup, but at some point the oldest version is gone. I am keen for a completely separate, long term snapshot of what I had that could be thrown in a cupboard, and any random family member clearing my house out as I get moved into a rest home at 108 can go through the photos and find a good one to put on my headstone.

I am also keen for protection against doing something dumb and losing everything (like losing my hard drive and finding out for some reason I can't access my backups because I lost the encryption key because I put it in bitwarden and they shut down years ago and I never moved the key over because I forgot it was stored there).

[–] [email protected] 3 points 1 week ago (3 children)

ZFS and BTRFS both provide that functionality. Have a look into the features.

load more comments (3 replies)
[–] [email protected] 5 points 1 week ago (1 children)

I believe M-Disc to be the best consumer grade, optical solution out there. If you want to go commercial grade you are looking for LTO tapes, but your costs begin rising exponentially. If M-Disc claims are to be believed, they should last well longer than your requirement and be able to handle your data footprint using multiple, but not an unreasonable amount of discs.

No matter which solution you choose, if you are targeting multiple decades, you must save not only the media, but ideally the drive, computer and software used to archive. There is no guarantee that any of the existing technology will be relevant or backward compatible across several decades.

load more comments (1 replies)
[–] [email protected] 5 points 1 week ago (1 children)

Just a hdd in usb caddy? IMHO good enough for 4 tier backup.

[–] [email protected] 4 points 1 week ago (1 children)

Yes this seems to be the general theme. Main issue is sorting out a file system. I can use a self-repairing one, to recover from long term storage issues, but then it likely won't work in Windows which it may need to if I want a layman to be able to access it. So still some refinement of the plan but it's coming together.

I've also decided to print some physical photos, aiming for 100 per year, and will put everything in a container together. The physical photos are for in case the container is lost for decades and the drives die, then there will at least be something.

[–] [email protected] 3 points 1 week ago

If you need something which can withstand some bitrot on single drive, just use par2. As long is filesystem is readable, you can recover files even if bit of data get corrupted

[–] [email protected] 4 points 1 week ago (1 children)

This is why I do my first-level of backups with rsnapshot. It backs up to the plain filesystem using rsync and uses hard-links to de-dup between backups. No special filesystem, no encryption, restore is just an 'rsync' away.

[–] [email protected] 2 points 1 week ago (1 children)

Yes my issue is that I seem to be replacing a drive somewhere every couple of years. I am keen for something that can be stored in a cupboard for years, preferably a decent chance at lasting decades.

[–] [email protected] 3 points 1 week ago (1 children)

Ah - I gotcha. That's some terrible luck with drives.

load more comments (1 replies)
[–] [email protected] 4 points 1 week ago (1 children)

For photos? Archival prints. As a bonus, you also get a cool album to reminisce later in life.

[–] [email protected] 3 points 1 week ago (2 children)

I don't want to sort through the 50k photos, and can't print videos. I'm hoping in 10 or 20 years I'll be able to feed it into AI to spit out all the best ones, then I'll consider it.

We do have photos printed, but only a very small percentage of the total.

load more comments (2 replies)
[–] [email protected] 4 points 1 week ago (6 children)

For local backups it depends on what you want to have:

  • The cheapest option is a usb or thumb drive. But you have to regularly plug it in and copy your backup on it.
  • The lazy option is to buy a NAS and configure a backup job that regularly creates a backup. Versioned, incremental, differentials and full backups are possible as is WORM to add a bit of extra security. You can configure a NAS to only turn on specified times, do a backup and then turn off again. This will increase protection against encrypting malware. WORM also helps in this case.
    Or just let it run 24/7, create backups every hour and install extra services on it like AI powered image analysis to identify people and objects and let it automatically tag your photos. Cool stuff! Check out QNAP and Synology or build a NAS yourself.
    A NAS can also be configured to present its content in a LAN by itself. Any computer will automatically connect to it if the access isn't secured by user/password or certificate.

I recommend buying a NAS.

load more comments (6 replies)
[–] [email protected] 3 points 1 week ago* (last edited 1 week ago) (5 children)

ZFS with automatic snapshots and scrubbing. This will keep as many and as old snapshots as your like. It'll ensure the files don't rot. It'll ensure the media doesn't die, so long as you have enough redundancy and you replace disks as they die. This is what I'd trust for long term storage because I think I understand how and why it works. It should last as long as I feed it disks. If I delete something, I should be able to restore it from a snapshot. The hardware doesn't need to be anything fancy. Just a Pi 4/5 with a couple of WD Elements would be fine. Could add more disks for more redudnancy. I'm running 2-disk residency.

You don't have to touch the software if it's not exposed to the Internet. Whatever works today on it will work 20 years from now, so long as the hardware works. A couple of spare Pis, SD cards and power supplies should let it last for decades.

load more comments (5 replies)
[–] [email protected] 3 points 1 week ago* (last edited 1 week ago) (4 children)

I decided instead to use ZFS. Better protection than just letting something sit there. Your backups are only as good as your restores. So, if you are not testing your restores, those backups may be useless anyway.

ZFS with snapshots, replicated to another ZFS box. The replicated data also stores the snapshots and they are read-only. I have snapshots running every hour.

I have full confidence that my data is safe and recoverable.

With that said, you could always use M-disk.

load more comments (4 replies)
[–] [email protected] 2 points 1 week ago (2 children)

I'm thinking of using a HDD and keeping it at work, which is climate controlled. I'd bring it back every few months to sync the latest.

Since it's constantly being used, I'm pretty confident it'll be usable as a backup if my NAS fails, so it only needs to be "shelf stable" for a few months at a time. If you're retired or something, a safe deposit box at your local bank should do the trick.

[–] [email protected] 5 points 1 week ago (1 children)

If it's powered off, you'll have no idea when it dies. And they do die just sitting there.

I've actually had more failures of drives sitting around than ones running constantly.

load more comments (1 replies)
load more comments (1 replies)
load more comments
view more: next ›