I use Debian stable for my main OS for the stability, security and infrequent updates, and run all of my services in Docker containers to keep everything up to date.
this post was submitted on 05 Feb 2025
100 points (96.3% liked)
Selfhosted
41954 readers
476 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
I switched away from truecharts once scale switched to native docker and my experience has been much smoother since. TC had some kind of breaking change every other month, now I only have to worry about breaking changes when the actual apps have a major update.
The transition was way easier than i expected. First I set up nginx pointing to the TC load balancer for every url, so I could swap apps one at a time. Then I used heavyscript to mount the volumes for an app and rsynced them to a normal dir. With that I could spin up the community apps version or a custom docker config and swap over nginx once I confirmed it was working.
I use debian, so what's to keep up with? Apt upgrade is literally everything I need. My home server doesn't take a lot of my time except when I want to tweak something or introduce something new. I dont really follow all the trendy stuff at all and just have it do what I need.
I run proxmox on the host with docker in a VM for 90% of my stuff, OS updates I do like every 6 months maybe, I've done 1 major version upgrade on proxmox with no issues at all.
The docker containers auto-update via Komodo, and nothing really ever breaks anymore other than the occasional container error that needs a simple fix.
Everything important is backed up nightly using both proxmox backup server, and to backblaze B2 with restic.
I've never heard of komodo, I've heard a lot about Watchtower but I found it more annoying to set up due to its labeling systems. Is there any added benefit for Komodo over using a standard watch tower setup?
I haven't set up either of them, but my main concern is having a breaking change be automatically updated
Komodo is a full management setup, similar to Portainer, Dockge, etc.. It works reasonably well.
Watchtower doesn't require any labeling unless you want to exclude a container.
but my main concern is having a breaking change be automatically updated
Pinning to a major version usually solves this, ie; instead of using postgres:latest use postgres:14 which will give you updates only from version 14.
But also have backups in place, worst case you just roll back to before it updated.
Oh ok, thank you, I already use Portainer for my existing setup so it wouldn't make much sense to fully rework it. I haden't thought of version pinning though so I may implement that instead, it makes sense "breaking changes" wouldn't happen within the same major version.
Yeah pinning is great, you'll still need watchtower for auto updates too
Yea for sure, I plan to implement that as well when I have some free time.
I've never used true nass, but I've never had any issue with keeping up with releases. I use a proxmox host with Debian containers mostly, and then I use ansible to do any major changes to the hosts such as replacing certificates or upgrading the packages
Being said my backup structure isn't the most professional, I have a 8 TB external drive that I keep plugged in via USB and I have proxmox backup server on the same host and it creates backups nightly
This is why I'm still using a Synology ¯\(ツ)/¯
I can install all the fun stuff I want in Docker, but for the core OS services, it's outsourced to Synology to maintain for me
You might want to think about running a “stable” or “LTS” OS and spin up things in Docker instead. That way you only have to do OS level updates very rarely.
Thanks for this. I've recently been recreating my home server on good hardware and have been thinking it's time to jump into selfhosting more stuff. I've used Docker a bit, so I guess I'll have to do it the right way. It's always good to know what choices now will avoid future issues.
load more comments
(1 replies)
I run a Fedora server.
All of my apps are in docker containers set to restart unless stopped by me.
Then I run a cron job that is scheduled at like 3 or 4am that runs docker pull on all containers and restarts them. Then it runs all system updates and restarts the server.
Every week or so I just spot check to make sure it is still working. This has been my process for like 6 months without issue.
Try watchtower instead of cron jobs
Depends on your stance on risk since WatchTower has to run as privileged
This is a good point. Generally if can accomplish what I want with my own scripts, I will go that route. I'll probably avoid adding additional software to the mix since what I have works fine enough.
I'll check it out! Thanks!
OS updates I only bother with every 6-12mo, though I also use debian which doesn't push major updates all that regularly.
As far as software goes; pretty much everything is in a docker container with watchtower automatically pulling new updates to those nightly at 4am. It sends me email notifications, so It'll tell me if an update fails; combined with uptime-kuma notifying me if any of my services is unavailable for whatever reason.
The rest I'll usually do with the OS updates. Just because an update was released, doesn't mean you've gotta drop everything and install it right this moment.
You can choose a slower train for scale. Go for the stable release or even the enterprise release. Update once in a few months or so.
I went with Talos OS for my apps after the mess from IX-systems and for the most part it has been set and forget.
Do you run Talos on bare metal or on something like Proxmox? Care to discuss your k8s stack?
Currently I run Talos on a VM on scale. I went with Truecharts. The plan for me is to run it on bare metal at some point.
I’m looking at Talos on my Proxmox cluster as VMs. I’m trying to automate it all through ansible and currently stuck trying to bootstrap my secrets manager. Somewhat of an analysis paralysis at the moment. Thinking of using a cloud hosted one with some kind of a local passthrough cache in case the WAN connection gets disrupted.
In life? Amphetamines.
Release: stable
Keep the updates as hands off as possible. Docker compose, TTeck's LXC updater, automatic upgrades.
I come through once a week or so to update the stacks (dockge > stack > update), I come through once a month or so to update the machines (I have 5 total). Total time updating is 3hrs a month. I could drop that time a lot when I get around to writing some scripts to update docker images, then I'd just have to "apt update && apt upgrade"
Minimise attack surface and outsource security. I have nothing at all open to the internet, I use Tailscale to create tunnels. I'm trusting my security to Tailscale but they are much, much, better at it than I am.
Automatically upgrading docker images sounds like a recipe for disaster because:
- could pull down change that requires manual intervention, so things "randomly" break
- docker holds on to everything, so you'd need to prune old images or you'll eventually run out of disk space; if a container is stopped, your prune would make it unbootable (good luck if the newer images are incompatible with when it last ran)
That's why I refuse to automate updates. I sometimes go weeks or months between using a given service, so I'd rather use vulnerable containers than have to go fix it when I need it.
I run OS updates every month or two, and honestly I'd be okay automating those. I run docker pulls every few months, and there's no way I'd automate that.
I've encountered that before with Watchtower updating parts of a serrvice and breaking the whole stack. But automating a stack update, as opposed to a service update, should mitigate all of that. I'll include a system prune in the script.
Most of my stacks are stable so aside from breaking changes I should be fine. If I hit a breaking change, I keep backups, I'll rebuild and update manually. I think that'll be a net time save over all.
I keep two docker lxcs, one for arrs and one for everything else. I might make a third lxc for things that currently require manual updates. Immich is my only one currently.
Watchtower
Glad it works for you.
Automatic updates of software with potential breaking changes scares me. I'm not familiar with watchtower, since I don't use it or anything like it, but I have several services that I don't use very often, but would suck if they silently stopped working properly.
When I think of a service, I think of something like Nextcloud, Immich, etc, even if they consist of multiple containers. For example, I have a separate containers for libre office online and Nextcloud, but I upgrade them together. I don't want automated upgrades of either because I never know if future builds will be compatible. So I go update things when I remember, but I make sure everything works after.
That said, it seems watchtower can be used to merely notify, so maybe I'll use it for that. I certainly want to be around for any automatic updates though.
It's Watchtower that I had problems with because of what you described. Watchtower will drop your microservice, say a database, to update it and then not reset the things that are dependent on it. It can be great just not in the ham fisted way I used it. So instead I'm going to update the stack together, everything drops, updates, and comes back up in the correct order
Uptime Kuma can alert you when a service goes down. I am constantly in my Homarr homepage that tells me if it can't ping a service, then I go investigating.
I get that it's scary, and after my Watchtower trauma I was hesitant to go automatic too. But, I'm managing 5 machines now, and scaling by getting more so I have to think about scale.
I don't use Watchtower myself for the same reasons described, but I was under the understanding if you had a container as a dependency on another container that if you took the dependency down it also took the container down. Is this not actually true?
I am not the person to be asking, I am no docker expert. It's is my understanding depends_on: defines starting order. Once a service is started, it's started. If it has an internal check for "healthy" I believe watchtower will restart unhealthy containers.
This is blind leading the blind though, I would check the documentation if using watchtower. We should both go read the "depends on" documents as we both use it.
Strangely it sounds like that's correct. I was under the understanding that depends_on cared about it past start as well but it does not. It doesn't look like there's a native way of turning containers that are depending on one another when you turn the dependency off. It looks like the current recommended way of doing it is either with a Docker compose file (which doesn't help if the process crashed/was concidered unhealthy), or having a third party script on the host monitor the dependencies and if one is considered offline, it turns the dependees off.
Looking into it the concern has been approached twice now on the GitHub page, however every time that it's been brought up it's been closed for stale because nobody ever replies to the question
That was my conclusion as well, however I am at work and it's not appropriate to be reading docker documentation. Thank you for the write up.
load more comments
(1 replies)
Debian, baby.
For one I don't use software that updates constantly. If I had to log in to a container more than once a year to fix something, I'd figure out something else. My NAS is just harddrives on a Debian machine.
Everything I use runs either Debian or is some form of BSD
load more comments
(1 replies)
I use NixOS so if an update breaks, I just roll back. And since it's effectively a rolling release distribution there isn't any risk of being left behind on an outdated version.
Same here. I spent last month transitioning all my servers to NixOS and it feels so comfy! I do a small test on my desktop when I do something that might break stuff first, and then add it to server's config later.
--target-host and --use-remote-sudo makes it even better too.
If it works, I don't update unless I'm bored or something. I also spread things out on multiple machines, so there's less chance of stuff happening like you describe with the charts feature going away. My NAS is pretty much just a NAS now.
You can probably backup your configs/data, upgrade, then deploy jellyfin again, restore, and reconfigure. You should probably backup your data on your ZFS pool. But, I recently updated to the latest TrueNas Scale from ~5 year old FreeBSD version of TrueNas and the pools still worked fine (none of the "apps" or jails worked, obviously). The upgrade process even ported my service configurations over. I didn't care about much of the data in the pools, so only backed up the most important stuff.
load more comments
(1 replies)
First off, backups of the configs any user data that you can't torrent should the inevitable happen.
Then set time aside to do updates, I spend Wednesday evenings updating and improving my setup.
Then find a way to track update announcements, I use both an RSS reader and newrealeases.io to know when something I run gets an update
Just subscribe to the release channel. That varies from OS to OS or Software, but is worth it.
Use tools that are universal. For example, I have not used TrueNAS Scale because they did not support native docker at the time. OS specific solutions are more likely to break then universal once (truecharts vs docker)
To get up and running again after a complete failure i can just download the latest config and data from my backup and set up any distro that supports docker and my system is running again.
I do OS upgrades when they are available, usually within 1 or 2 days and containers are updated with watchtower daily.
I dont :) Mostly.
Honestly I have an auto backup system. And then set it up to auto update periodically. Then use Debian Server as it almost never breaks as a server distro.
Constant maintenance? What's that?
Here's my setup:
- OS - openSUSE Leap - I upgrade when I remember
- software - Docker images in a docker compose file; upgrading is a simple docker command, and I'll only do it if I need something in the update
- hardware - old desktop; I'll upgrade when I have extra hardware
I honestly don't think about it. I run updates when I get to it (every month or so), and I'll do an OS upgrade a little while after a new release is available (every couple years?). Software gets updated periodically if I'm bored and avoiding more important things. And I upgrade one thing at a time, so I don't end up with everything breaking at once. BTRFS snapshots means I can always roll back if I break something.
I don't even know what TrueCharts is. Maybe that's your issue?
view more: next ›