Re-thinking the Homelab Setup

The setup as described in the prior posts has actually served me well for the year or so it was running. During that time, though, I kept noticing things that bothered me, some to a larger extent than others. In no particular order, I found troubling that:

• Using OCI images simply shifts who I depend on. Distro packager, OCI packager, what is the functional difference? That the application is packaged by the developer? And how often does the developer have any background in system integration? To cite an example, I managed to horribly break my MySQL database during a planned update because the container startup timed out in the middle of the data dictionary update. Hooray for backups, although I did resuscitate the broken data files as an exercise in worst-case recovery. Running health checks while a DB is updating is bad.
• Docker is a source of needless complexity, although I should be fair enough to state that the source of the complexity is not solely within Docker. Process, user and network namespaces (and I am sure I forget a few things) need to be managed, multiple file systems need to be overlaid to make a final running container. Depending on configuration there might be hidden bridge networks to allow certain kinds of communication. The documentation of all those features is, to put it carefully, modest. The runtime implements name resolution, takes it upon itself to configure the host's firewall and network interfaces, and is in general intransparent to the administrator. System administration with Docker shifts from "I know how I configured my system, why I did so, and how I accomplished the goal" to "I configured Docker and trust the application to do the right thing with that configuration."
• Secrets management. Unless you run a Docker cluster (Stack? The terminology is... dubious), Docker has no facilities to manage secrets. They need to be placed in Compose files, or else loaded from the file system in some fashion, with no obvious best practice. For a homelab system, this may not matter too much, but I prefer not to have to worry about accidentally exposing an important password while sharing configuration. To be fair, this is an un-solved problem, and I see no easy solution here.
• Volume management. I found this to be an absolutely horrible experience. Sure, it's nice that my container resets to a "default" state every time I restart it, but some things need persistence. How do I handle this? Do I bind-mount a host directory into the container? Set up a named volume? How about a situation where multiple containers need to access a volume for whatever reason? Certificates? Again, the process is complex when it really does not need to be.

On top of Docker, there is the fact that Linux - and at this point, I hesitate to call it GNU/Linux - is a fast-moving target. There is a reason enterprise distributions are as far behind as they are in terms of software versions, and rolling-release distributions exist. How things are done changes rapidly and constantly. Do not misunderstand me, progress is a good thing, but I like the systems that provide services to be stone boring and just work.

All that in mind, I decided to give FreeBSD a shot, and quickly discovered the system does what I want it to in ways I could never convince Linux+Docker to support. Jails simplify container management substantially, although they do not solve every problem - most notably, they do not solve secret management. Between bhyve, VNET jails and FreeBSD's networking stack, I am finding managing network connectivity far easier than I did with Docker. Volume management uses fstab files for both jails and the host. ZFS makes assigning volumes to jails and VMs easy. Sharing volumes similarly is easy enough. Native support for NFSv4 ACLs makes for much nicer rights management than the bolted-on POSIX ACLs of Linux. And I don't need a moloch of an application to manage all this - jails are natively supported by the OS and userland. The choice was simple - when it came time to rework my homelab, I decided to work from a FreeBSD base.

That is not to say there are no rough spots. Not every application is supported through packages and ports - manual compilation is often required, including reverse-engineering Dockerfiles. In particular NodeJS applications do not always play nice with FreeBSD - it's far from a first-class citizen in web development. I'm looking at you, Authentik, as prime example for both.

I will be journaling the setup of this incarnation of my homelab in this blog. Hope you all enjoy the ride.