Dogfooding it, Pt 4 - Traefik

Traefik is a reverse proxy with container integration. With properly set up containers and networking, it's minimal configuration to get it working in this lab - much easier than nGinx.

Our directory structure for this consists of a handful of directories. We do need the Traefik UID to be defined in the host OS, as we grant permission to access the Docker socket.

$ useradd -G docker -d /srv/data/docker/traefik -s /bin/nologin -u 10001 traefik
$ mkdir -p /srv/data/docker/traefik/{acme,log,rules}

We don't secure these directories just yet, because we still have some work to do in them.

Again, we'll be embedding the configuration in a docker-managed config. This file can be arbitrarily names, but let's just call it traefik.toml

[global]
checkNewVersion = true
sendAnonymousUsage = false

[entryPoints.web]
address = ":10080"

[entryPoints.web.http.redirections.entryPoint]
to = ":443"
scheme = "https"

[entryPoints.websecure]
address = ":10443"

[entryPoints.websecure.http3]
advertisedPort = 443

[entryPoints.websecure.http.tls]
options = "tls-opts@file"
certResolver = "le"

[certificatesResolvers.le.acme]
email = "<your email>"
storage = "/acme/acme.json"
caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
tlsChallenge = true
preferredChain = "ISRG Root X2"
keyType = "EC384"

[log]
level = "INFO"
filePath = "/logs/traefik.log"

[accesssLog]
filePath = "/logs/access.log"

[accessLog.filters]
statusCodes = [ "204-299","400-499","500-599" ]

[api]
dashboard = true

[ping]
entryPoint = "traefik"

[providers.swarm]
endpoint = "unix:///run/docker.sock"
exposedByDefault = false
network = "homelab"

[providers.file]
directory = "/rules"
watch = true

Let's take it from the top. I want to be notified of updates, but I don't want to send telemetry. Feel free to change those values if you prefer them set differently. Our unencrypted entry point will be port 10080, and have a global redirect to port 443 set.

Our encrypted entrypoint will be on port 10443, but advertise port 443 on http3. The high ports allow us to run Traefik as a non-root user. TLS for our encrypted entrypoint uses options configured in the dynamic file tls-opts, and the certificate resolver "le",

That resolver sends the CA an email address, stores its stuff in /acme/acme.json, and uses the TLS-ALPN-01 challenge for domain validation. The configuration file points Traefik to use the staging server at Let's Encrypt; simply comment that line out and regenerate the configuration once you want to use the production server.

We wamt tje ECDSA-only chain of trust, and to use an ECC key with a length of 384 bits.

Logs are stored in /logs, with the access log filtering errors. We really only want to see successful hits. The dashboard is enabled.

The ping endpoint is useful for health checks. Not something we're setting up at this stage, but it doesn't hurt to have and will be useful down the line.

Finally, our configuration providers are the Docker swarm, although containers we detect are not proxied by default - we need to mark them explicitly. The default network to communicate with containers, unless we specify differently in the container, is homelab.

File-based configuration is stored in /rules, and is monitored and reread every 30 seconds by Traefik.

Next, create /srv/data/docker/traefik/rules/tls-opts.toml with the following content.

[tls.options.tls-opts]
minVersion = "VersionTLS12"
cipherSuites = [
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256",
"TLS_AES_256_GCM_SHA384",
"TLS_CHACHA20_POLY1305_SHA256",
"TLS_FALLBACK_SCSV"
]
curvePreferences = [
"CurveP384",
"X25519"
]
sniStrict = true

It's 2024. We use TLS1.2 minimum, and restrict our ciphers to highly secure ones. No reason to muck with legacy ciphers. Similarly, we expect Server Name Indication to be implemented properly on our clients.

While we're there, set up another file, /srv/data/docker/traefik/traefik-allowlist.toml. We'll need it later.

[http.middlewares.traefik-allowlist.ipAllowList]
sourcerange = [ "127.0.0.1/32","10.0.0.0/8" ]

This sets up IP whitelisting for the local host and all IP addresses starting with 10. As that is a private subnet, and we are using it, connections from there should be safe. Feel free to be more restrictive.

Now we can lock down our persistent storage for this contanier.

$ sudo chown -R traefik:traefik /srv/data/docker/traefik
$ sudo chmod -R g-rwx,o-rwx /srv/data/docker/traefik

Now we're ready to bring the container online. First, we need to set up port forwarding on our firewall - we'll need it for TLS-ALPN. So, after connecting to the firewall with SSH:

sudo firewall-cmd --permanent --add-port-forward=port=80:proto=tcp:toaddr=10.32.0.2
sudo firewall-cmd --permanent --add-port-forward=port=443:proto=tcp:toaddr=10.32.0.2

This adds global port forwarding rules to the firewall, to send all packets coming in on ports and 80 and 443 to the home lab server.

Next, deploy the Traefik container

$ docker config create traefik traefik.toml
$ docker service create \
--config "source=traefik","target=/etc/traefik/traefik.toml" \
--label "traefik.enable=true" \
--label "traefik.http.routers.traefik-rtr.entrypoints=websecure" \
--label "traefik.http.routers.traefik-rtr.rule=Host(\`traefik.<your domain>\`)" \
--label "traefik.http.routers.traefik-rtr.service=api@internal" \
--label "traefik.http.routers.traefik-rtr.middlewares=traefik-allowlist@file" \
--label "traefik.http.services.dummy-svc.loadbalancer.server.port=9999" \
--network homelab --network homelab-bridge --name traefik \
--publish 80:10080 --publish 443:10443 \
--mount "type=bind","source=/srv/data/docker/traefik/rules","target=/rules" \
--mount "type=bind","source=/run/docker.sock","target=/run/docker.sock","readonly=true" \
--mount "type=bind","source=/srv/data/docker/traefik/acme","target=/acme" \
--mount "type=bind","source=/srv/data/docker/traefik/logs","target=/logs" \
traefik:3.0

Yeah. It's that kind of command. We mount our configuration into the container, and set a bunch of labels. Specifically, in order, we mark the container as enabled for reverse proxying through Traefik, with the entrypoint set to the secure port, and answering to the hostname "traefik" on your DNS domain. The running service is a magic string telling Traefic this is its internal web services - dashboard and API. We allow only the IP addresses defined in the allowlist earlier to access this server. The dummy port is required, since Traefik cannot derive port information from a Docker swarm. It is not used. The server is on the networks homelab and homelab-bridge, named "Traefik", and publishes ports 10080 and 10443 to ports 80 and 443 on the container host. We bind-mount the rules, log and acme directories from persistent storage into the container and allow it access to the Docker socket. That socket is mounted read-only - Traefik needs to get information, not to be able to give orders to Docker. Finally, we specify we want version 3.0.x.

Traefik sometimes breaks things between minor versions, so we nail it to the one our configuration is tested with.

Now, we need to label our Ghost instance.

docker service update --label-add "traefik.enable=true" --label-add "traefik.http.routers.ghost-rtr.entrypoints=websecure" --label-add "traefik.http.routers.ghost-rtr.rule=Host(\`ghost.turriff.net\`)" --label-add "traefik.http.routers.ghost-rtr.service=ghost" --label-add "traefik.http.services.ghost.loadbalancer.server.port=2368" ghost

Same as for the internal API, but we inform Traefik that Ghost listens on port 2368. And that's it, there is now a working blog.