Homelab IaC

Deploy apps as code across two platforms: Unraid server(s), where apps run as Docker Compose stacks, and Proxmox, where Terraform creates an LXC and the app is installed into it natively. Ansible is the deployment tool on both. Secrets are pulled from HashiCorp Vault at deploy time; CI/CD is Forgejo Actions once Forgejo itself is running.

There's also a third piece of infrastructure alongside those two: a bare-metal k3s cluster (homelab-utils) across 4 Raspberry Pis, bootstrapped by Ansible too. It doesn't fit the src/<app>/ table above the way Unraid/Proxmox apps do — Terraform never provisions it, and no stacks:/apps: list drives what's on it — so it gets its own inventory group and role instead. It does run shared services deployed via Helm, though: k3s_monitoring, k3s_metallb, and the cluster's own shared Postgres (k3s_postgres, a third platform for src/shared/postgres/). See playbooks/k3s.yml and "K3s (Raspberry Pi)" below.

See CLAUDE.md for the short version of the architecture decisions.

Layout

src/
  <app>/
    common/vars.yml        config that holds on either platform
    ansible/
      unraid/              docker-compose.yml + .env.example + platform overrides
                           (+ optional icon.png for the Unraid Docker page)
      proxmox/             install.yml + config/systemd templates + platform overrides
      kubernetes/          vars.yml + *.yaml.j2 manifests (normally a HelmChart CR)
    terraform/             this app's LXC definition (Proxmox only)
  shared/<service>/        same shape, for services multiple apps use
build/config/
  ansible/
    inventory/
      hosts.yml            static: unraid_servers, proxmox_nodes, k3s_cluster
      proxmox.yml          dynamic: proxmox_guests, from the Proxmox API
      group_vars/all.yml   Vault connection, Compose Manager project path
      group_vars/k3s_*.yml k3s cluster name/version, per-role vars (server/agent)
      host_vars/<host>.yml which stacks/apps deploy there
                           (k3s apps are in group_vars/k3s_cluster.yml instead)
    roles/compose_stack/   Unraid: sync compose + render .env + provision DB + up
                           (deploy.yml / remove.yml, picked by the entry's state:)
    roles/lxc_app/         Proxmox: install natively + render config + systemd
    roles/k3s_app/         k3s: render app manifests + Secret + provision DB
                           (deploy.yml / remove.yml, picked by the entry's state:)
    roles/pve_backup/      Proxmox node: NAS backup storage + vzdump schedule
    roles/k3s_node/        k3s Pi: cgroups/swap prep + install server or agent
    roles/k3s_monitoring/  k3s control plane: lean kube-prometheus-stack via HelmChart CR
    roles/k3s_metallb/     k3s control plane: MetalLB (LoadBalancer IPs) via HelmChart CR
    roles/k3s_postgres/    k3s control plane: shared Postgres (CloudNativePG) via HelmChart CR
    roles/k3s_cert_manager/ k3s control plane: cert-manager + ClusterIssuer via HelmChart CR
    roles/unattended_upgrades/  any apt host: hands-off patching, no auto-reboot
    roles/k3s_maintenance/ k3s Pi: drain + reboot-if-required + uncordon, one at a time
    playbooks/
      deploy.yml           everything (Unraid + Proxmox — not k3s, see below)
      unraid.yml           just Unraid
      proxmox.yml          just the Proxmox guests
      pve_host.yml         just the Proxmox node (backups)
      k3s.yml               bootstraps the k3s cluster (manual-only, not in deploy.yml)
      k3s_maintenance.yml   rolls pending reboots across it, serial: 1 (manual-only)
  terraform/               Proxmox provider, pg state backend, LXC module calls
.forgejo/workflows/      CI/CD (needs a self-hosted runner, see below)
docs/vault-secrets.md    Vault path/key reference
docs/authentik-migration.md  hand-run cutover: Authentik from Unraid to k3s

Terraform's scope stops at the guest — it creates the LXC and nothing more, because the Proxmox provider has no way to install an application. Ansible picks up from there, which is why it's the common tool and Terraform is Proxmox-only.

How config is layered

An app's settings are split by whether they survive a change of platform:

• src/<app>/common/vars.yml — version, ports, domain, database name/user.
• src/<app>/ansible/<platform>/vars.yml — the rest. Data paths differ (/mnt/user/appdata/forgejo vs /var/lib/forgejo), and so does how the database is reached (the shared-postgres Docker network alias only exists on Unraid).

Both roles merge them the same way: common, then platform overrides, then Vault secrets last. Nothing is restated between platforms.

How a deploy works

Unraid. For each stack in a host's stacks: list, compose_stack:

1. Copies src/<stack>/ansible/unraid/docker-compose.yml to the host's Compose Manager project folder, unmodified — it's static and only references ${VAR}.
2. Merges the two vars.yml layers with the stack's Vault KV path and renders the project's .env. Nothing secret ever touches git.
3. If the stack declares a db: block, ensures its database/role exist on the shared Postgres instance (idempotent — safe to add new apps to the shared service over time without touching existing data).
4. Copies icon.png, if the stack ships one, to both the project folder and /mnt/user/appdata/icons/ (see "Icons on the Unraid Docker page" below).
5. Runs docker compose up -d for the stack.

Proxmox. terraform apply creates the LXC first. Then, for each app in a guest's apps: list, lxc_app does the same config merge and database provisioning, runs that app's ansible/proxmox/install.yml, and manages its systemd unit instead of a Compose project.

k3s. For each app in k3s_apps: (group_vars/k3s_cluster.yml, not host_vars — an app belongs to the cluster, not a node), k3s_app:

1. Does the same two-layer config merge, using src/<app>/ansible/kubernetes/vars.yml as the platform layer.
2. Provisions the app's database/role on the CloudNativePG cluster if it declares a db: block — the same community.postgresql tasks the other two platforms use, pointed at MetalLB's LoadBalancer address for the CNPG primary.
3. Renders the app's Vault path into a Kubernetes Secret (plus the Namespace), keys passed through verbatim.
4. Renders every *.yaml.j2 the app ships into /var/lib/rancher/k3s/server/manifests/, where k3s's own deploy and helm controllers pick them up.

Nothing here talks to the Kubernetes API — no helm, no kubeconfig, no kubernetes.core. Ansible writes files; k3s reconciles them. The consequence worth remembering: a green run means the manifests landed, not that the workload came up. Check with kubectl -n <namespace> get pods.

Secrets land on disk in the rendered .env (root-readable, on the Unraid box) or, on k3s, in a 0600 manifest and then in etcd as an ordinary Kubernetes Secret — this is "fetch from Vault at deploy time," not a zero-secrets-at-rest model. Fine for a homelab; revisit if that changes.

Icons on the Unraid Docker page

There are two icons per stack, and they come from completely different places. Commit one src/<app>/ansible/unraid/icon.png and the deploy feeds both. A stack running more than one container can commit icon-<service>.png alongside it for the extra services — see "More than one container" below.

The stack row

Compose Manager draws the collapsible header row for the whole project, and serves its icon straight off disk from icon.png in the project folder — /boot/config/plugins/compose.manager/projects/<stack>/icon.png, the same directory the compose file and rendered .env land in. No label, no template, no URL: the file is there or the row has no icon. .jpg, .gif, .svg and an extensionless icon also work, since the plugin serves the file itself rather than handing a path to the webgui — but stick to PNG so the same file can serve the labels below.

This is a feature of Compose Manager Plus (mstrhakr/compose_plugin), the maintained fork. The original Docker Compose Manager plugin (dcflachs, archived April 2026) has no icon support at all — on that one the copied file is simply ignored, so the deploy is safe either way, you just won't see it.

The containers under it

Unraid's Docker page gets each container's icon, WebUI link and console shell out of the dockerMan template that created it, under /boot/config/plugins/dockerMan/templates-user/. A Compose stack never creates one, which is why compose-deployed containers show a question mark and no WebUI entry. Unraid 6.10+ falls back to Docker labels when there's no template, so each service sets them itself:

labels:
  net.unraid.docker.icon: ${STACK_ICON}
  net.unraid.docker.webui: "http://[IP]:${FORGEJO_HTTP_PORT}/"
  net.unraid.docker.shell: sh

They're per service, not per stack — a stack with two containers labels each one separately, and can point them at different icons.

• icon — ${STACK_ICON} is set by the role, not by hand. The same committed icon.png is copied a second time to /mnt/user/appdata/icons/<stack>.png and the label points there; ship no icon.png and the variable renders empty, leaving the placeholder. To use a hosted image instead, set STACK_ICON in the app's ansible/unraid/vars.yml — it's merged over the role's value.

  A second copy rather than reusing the project-folder one because this label is a path the webgui resolves on every Docker page load, and that path shouldn't run through the flash drive.

  PNG only here — unlike the stack row, this goes through the webgui's own icon handling, where SVG renders as the question-mark fallback and WebP renders nothing at all. A remote URL is fetched when the page renders, so an unreachable host breaks the icon — the reason a committed file is the default.

• webui — [IP] is substituted with the host's address. The port must be the published one, so interpolate the stack's own port variable rather than the container port. Omit the label entirely for something with no web interface (as shared/postgres does) — an empty value still draws a WebUI entry that goes nowhere.

• shell — sh or bash, whichever the image actually has. Alpine-based images (Forgejo) need sh; Debian-based ones (Postgres, Shelfarr) can take bash.

One caveat: editing a compose container's labels through the Unraid UI writes a template for it, and the template then wins over the labels until you clear it. Change these in the compose file and redeploy, not in the webgui.

More than one container

${STACK_ICON} is a single value — the stack's icon.png — so a stack with two services would otherwise have to give both the same image. Since the labels are per-service, commit icon-<service>.png next to icon.png for each additional container, named for the compose service:

src/arr/ansible/unraid/
  icon.png              → ${STACK_ICON}           → /mnt/user/appdata/icons/arr.png
  icon-prowlarr.png     → ${STACK_ICON_PROWLARR}  → /mnt/user/appdata/icons/arr-prowlarr.png

The variable is the service name uppercased with - folded to _ (icon-shelfarr-libation.png → ${STACK_ICON_SHELFARR_LIBATION}). These get the appdata copy only — there is one Compose Manager stack row and icon.png already has it — and, like STACK_ICON, each can be overridden with a hosted URL by setting the same name in the app's ansible/unraid/vars.yml.

The variable only exists when the file does. Interpolating ${STACK_ICON_FOO} with no icon-foo.png committed gets Compose's unset-variable warning and the question-mark placeholder — so if the icon is meant to be a URL, set it in vars.yml rather than relying on the file.

Backups (Proxmox)

Proxmox guests keep all their state on their own rootfs. Nothing is bind-mounted in from the NAS, which is the opposite of the obvious instinct — the reason is that vzdump deliberately excludes bind mounts, so a bind-mounted repository directory would be the one thing missing from the backup that was made to protect it. Keeping everything local means one archive is a complete copy of the guest's filesystem.

playbooks/pve_host.yml (role: pve_backup) configures the off-box copy on the node: an NFS storage pointing at the NAS, and a scheduled vzdump job writing to it. Retention is the storage's prune-backups settings. Point it at a Proxmox Backup Server instead by setting pve_backup_storage_type: pbs in host_vars/<node>.yml.

Data that lives outside a container is that app's problem, not vzdump's. Forgejo's database is on the shared Postgres over on Unraid, so restoring its container alone would give back every repository with no issues, pull requests, users or permissions. The fix is a pg_dump on a systemd timer inside the container, writing into its own filesystem — which puts the dump inside the same archive as the repositories it belongs to. That only works if it finishes before the backup window, so the two schedules are a pair:

	Set in	Default
Forgejo database dump	src/forgejo/ansible/proxmox/vars.yml → FORGEJO_DB_DUMP_ONCALENDAR	01:30
vzdump job	inventory/host_vars/pve.yml → pve_backup_schedule	02:00

Any future app with external state should follow the same pattern. Restore steps are in src/forgejo/ansible/proxmox/README.md.

None of this covers the Unraid side, which has its own backup arrangements outside this repo.

Prerequisites

Both platforms

• Ansible control node needs to be Linux — on Windows, run it from WSL2, not natively.
• If this repo is checked out on a Windows mount (e.g. under /mnt/c/..., which is where a Seafile-synced folder ends up) rather than the native WSL filesystem, Ansible will warn that ansible.cfg is being ignored because the directory looks world-writable — that's DrvFs reporting 777 on everything, not a real permissions problem. Either export ANSIBLE_CONFIG="$(pwd)/ansible.cfg" before running (from the build/config/ansible/ dir), or fix it permanently by adding to /etc/wsl.conf inside WSL:

  [automount]
  options = "metadata,umask=22,fmask=11"
  

  then wsl --shutdown from PowerShell and reopen the shell.
• pip install ansible hvac psycopg2-binary
• ansible-galaxy collection install -r build/config/ansible/requirements.yml
• A reachable Vault instance, with secrets populated per docs/vault-secrets.md.

Unraid

• Root SSH enabled on the Unraid box(es) (Settings → Management Access).
• Compose Manager (Docker Compose plugin) installed on Unraid — already done.
• Python3 on each Unraid host itself, not just the controller: the compose_stack role's tasks (file, copy, template, docker_compose_v2) run on the target, and Ansible modules — unlike raw/ad hoc shell — need a Python interpreter there to execute at all. Stock Unraid doesn't ship one. Installing it by hand into the running system doesn't survive a reboot, same reason as the SSH-key gotcha above — Unraid rebuilds its root filesystem from the flash drive on every boot. Install the NerdTools plugin (Community Applications → search "NerdTools", formerly "NerdPack") and enable python3 in its package list — it reinstalls whatever you've selected on every boot itself, same mechanism Compose Manager already relies on. ansible.cfg already sets interpreter_python = auto_silent, so Ansible finds it automatically wherever NerdTools puts it; no path to hardcode. Verify after installing:

  ansible unraid_servers -m ping
  

Proxmox

• Terraform CLI (>= 1.6) on whatever runs terraform apply.
• A Proxmox API token with rights to create containers, stored in Vault at homelab/ci/proxmox (see docs/vault-secrets.md). Both the Terraform provider and Ansible's dynamic inventory authenticate with it.
• A terraform_state database and terraform role on the CloudNativePG cluster on k3s (192.168.50.81), for backend "pg" — not on either shared Postgres this repo deploys, which would be circular. See "Bootstrapping" below.
• Container templates downloaded on the Proxmox node for whatever OS the LXCs are built from.
• An NFS export on the NAS for backup archives, reachable from the Proxmox node — set it in inventory/host_vars/pve.yml. See "Backups (Proxmox)".
• Root SSH from the Ansible controller to the Proxmox node itself, not just the guests: playbooks/pve_host.yml configures the node over SSH.
• Unlike Unraid, nothing special is needed inside the guests: they're ordinary Linux containers with Python already present, so Ansible works without the NerdTools workaround above.

K3s (Raspberry Pi)

• 4 Raspberry Pis running Ubuntu Server (64-bit), already imaged and on the network at the addresses in inventory/hosts.yml → k3s_cluster.
• An ansible user on each Pi with NOPASSWD sudo and this repo's k3s SSH key installed as an authorized key — see "SSH access" below. Not automated by this repo; create it by hand (or via cloud-init at image time) before the first run of playbooks/k3s.yml.
• homelab/k3s-homelab-utils → K3S_TOKEN set in Vault before the first run — see docs/vault-secrets.md. Both the server and every agent read this same fixed value rather than one generating it and handing it to the other, which is what makes a rebuild reproducible.
• Nothing else special: python3 ships with Ubuntu Server, and the k3s install script (get.k3s.io) handles containerd, the systemd unit, and everything else that isn't Pi-specific. The one Pi-specific thing — ensuring the memory cgroup controller is on — is handled by roles/k3s_node itself, not a prerequisite here.

Keeping it patched

playbooks/k3s.yml also applies roles/unattended_upgrades to every node — apt updates install themselves on their own daily schedule, no login required. It sets Unattended-Upgrade::Automatic-Reboot "false" though, so a kernel or containerd update that needs a reboot to take effect just sits applied-but-inactive until one happens; blindly auto-rebooting a k3s node takes its pods down with no warning.

playbooks/k3s_maintenance.yml is the other half: it checks /var/run/reboot-required on each node and, only where it's set, cordons and drains the node, reboots it, waits for it to report Ready again, then uncordons it — one node at a time (serial: 1), so patching the cluster never means dropping every workload at once. Manual-only for now, same as playbooks/k3s.yml:

ansible-playbook playbooks/k3s_maintenance.yml

Eventually this is meant to run on a schedule from a self-hosted Forgejo Actions runner rather than by hand — not wired up yet, see the header comment in the playbook.

Metrics (Prometheus)

playbooks/k3s.yml also deploys roles/k3s_monitoring: a lean kube-prometheus-stack (Prometheus + prometheus-operator + node-exporter + kube-state-metrics — no Grafana, no Alertmanager) so tools like OpenLens can show node/pod metrics. It's installed as a HelmChart custom resource dropped into k3s's own auto-deploying manifests directory (/var/lib/rancher/k3s/server/manifests/) rather than run through a helm binary — k3s's bundled helm-controller reconciles it the same way it installs its own Traefik and ServiceLB, so this needs no extra Ansible collection or kubeconfig on the controller. See roles/k3s_monitoring/defaults/main.yml for the resource sizing (tuned for a Raspberry Pi 4, not a real node) and why the control-plane component scrapers (kubeControllerManager, kubeScheduler, kubeProxy, kubeEtcd) are disabled — k3s doesn't expose those the way kube-prometheus-stack expects, so leaving them on just produces permanently-"down" targets.

To point OpenLens at it: open the cluster's Settings → Metrics, set Prometheus to "Auto detect" or explicitly to the Operator provider — it should find the kube-prometheus-stack-prometheus service in the monitoring namespace via the API server proxy, the same path OpenLens already uses to reach the cluster, so nothing needs to be exposed outside it.

LoadBalancer IPs (MetalLB)

playbooks/k3s.yml also deploys roles/k3s_metallb: MetalLB in L2 mode, handing out real LAN IPs to type: LoadBalancer Services instead of the ClusterIP-only world k3s would otherwise leave homelab-utils apps in. It's installed the same way as monitoring — a HelmChart CR for k3s's helm-controller to reconcile — plus a plain IPAddressPool/L2Advertisement manifest dropped in the same directory; k3s's deploy controller applies both kinds of file and retries the config manifest until the HelmChart's CRDs exist.

MetalLB replaces k3s's bundled ServiceLB (Klipper), it doesn't run alongside it — both would otherwise compete to satisfy the same Services. That's why inventory/group_vars/k3s_cluster.yml sets k3s_server_extra_args: ['--disable=servicelb'] — a separate, server-only var from k3s_extra_args, since k3s agent doesn't understand --disable and would fail to start if it were passed there too. The IP pool itself is k3s_metallb_address_range in the same file — a range on the cluster's LAN (192.168.50.0/24) that DHCP and every static assignment in inventory/hosts.yml steer clear of. There's no built-in default: the role fails fast if it's still empty.

Enabling --disable=servicelb on an already-running cluster doesn't need a separate step: roles/k3s_node notices its exec line has changed and reinstalls (restarting just the k3s service, not the node) the next time playbooks/k3s.yml runs — see CLAUDE.md → "Key decisions".

Postgres (CloudNativePG)

playbooks/k3s.yml also deploys roles/k3s_postgres: CloudNativePG (CNPG), a Postgres operator, installed the same way as monitoring and MetalLB — a HelmChart CR for k3s's helm-controller to reconcile — plus a plain Cluster CR (the actual database) and a Secret (superuser credentials), dropped in the same auto-deploying manifests directory. This is a third platform for src/shared/postgres/, alongside the Unraid Compose stack and the Proxmox placeholder: roles/k3s_postgres layers src/shared/postgres/common/vars.yml with src/shared/postgres/ansible/kubernetes/vars.yml the same way compose_stack/lxc_app layer an app's config, and pulls the superuser password from the same homelab/shared/postgres Vault path those platforms already use (see docs/vault-secrets.md) — a separate physical instance, not a shared login across platforms, just the same identity.

1 primary + 1 replica, kept off the control-plane Pi via nodeAffinity (see roles/k3s_postgres/templates/postgres-cluster.yaml.j2) — CNPG's own pod anti-affinity then spreads the two across the 3 worker Pis. Storage is k3s's default local-path StorageClass; that's node-local, but resilience here comes from CNPG's own streaming replication between instances, not from shared storage, so losing one instance's disk doesn't lose the data as long as the other instance is up. HA replication only for now — no ScheduledBackup to the NAS or anywhere else yet.

To connect: kubectl -n shared-postgres get svc lists four Services. Three are CNPG's own, all ClusterIP — shared-postgres-rw (the current primary), -ro (replicas only) and -r (any instance). An app running on the cluster uses the first of those by DNS and needs nothing else:

shared-postgres-rw.shared-postgres.svc.cluster.local:5432

The fourth, shared-postgres-lb, is a LoadBalancer on k3s_postgres_loadbalancer_ip (inventory/group_vars/k3s_cluster.yml — inside the MetalLB pool above), which is how the cluster is reached from outside it:

psql -h 192.168.50.81 -U postgres          # password: homelab/shared/postgres

It's declared in the Cluster CR as a CNPG managed service rather than as a Service manifest of our own, so the operator keeps its selector pointed at whichever instance is currently primary — a hand-written Service would need re-pointing by hand after a failover. Credentials are the shared-postgres-superuser-vault Secret Ansible renders; CNPG separately auto-creates shared-postgres-app for the default app database it bootstraps, which nothing uses yet.

The -vault suffix keeps that Secret clear of <cluster>-superuser, the name CNPG uses for the superuser secret it generates itself. Name our own object that and the operator treats it as one it already authored, so a rotated password reaches the Secret and never reaches the database — the Secret reads correctly, psql from the LAN keeps failing authentication, and only select rolpassword is null from pg_authid where rolname='postgres' inside the pod shows the disagreement.

That LoadBalancer is what makes the controller-reachable host:port the community.postgresql tasks need — the same ones compose_stack and lxc_app use to create an app's database and role — available here too, and roles/k3s_app now uses exactly those tasks for any app declaring a db: block.

Note the deliberate asymmetry, which reads like a mistake until you know why: provisioning uses 192.168.50.81 (the LoadBalancer) because it runs on the Ansible controller, which is off-cluster and can't route to a ClusterIP; apps use shared-postgres-rw.shared-postgres.svc.cluster.local because sending pod traffic out to the LAN and back would put MetalLB's L2 speaker in the path of every query for nothing. Two addresses, one database, each correct for its caller.

Without kubectl to hand, or if MetalLB is having a bad day, the in-cluster routes still work:

kubectl -n shared-postgres port-forward svc/shared-postgres-rw 5432:5432
kubectl -n shared-postgres exec -it shared-postgres-1 -- psql -U postgres

No backups. HA replication only — a streaming replica on another Pi, which covers a dead SD card and not a dropped table, a bad migration, or a mistyped DROP. This mattered less when the cluster held nothing; it matters now that Authentik's database lives here rather than on Unraid. A CNPG ScheduledBackup to the NAS is the obvious next piece of work and isn't built yet.

TLS (cert-manager)

playbooks/k3s.yml also applies roles/k3s_cert_manager: cert-manager plus one ClusterIssuer named letsencrypt, both via the usual HelmChart CR + plain manifest pair. An app's Ingress opts in with a single annotation —

annotations:
  cert-manager.io/cluster-issuer: letsencrypt

— and cert-manager creates and renews the certificate on its own.

Two values have no default and the role refuses to run without them, both in inventory/group_vars/k3s_cluster.yml:

• k3s_cert_manager_acme_email — an ACME account is registered against it.
• k3s_cert_manager_solver — the DNS-01 stanza for your DNS provider, rendered into the issuer as-is. A commented Cloudflare example is in that file; for anything else take the stanza from cert-manager's docs. Its API token goes in Vault at homelab/k3s-cert-manager.

DNS-01 rather than HTTP-01 because HTTP-01 needs Let's Encrypt to reach this cluster from the internet on port 80, which it can't. That has a useful consequence: a certificate can be issued before DNS points at the cluster, so a migration's TLS is settled before its cutover rather than after.

While working out a solver, point k3s_cert_manager_acme_server at Let's Encrypt staging — production allows five failed validations per hostname per hour, and exhausting it means waiting, not retrying. Check it registered:

kubectl get clusterissuer letsencrypt -o jsonpath='{.status.conditions[*].message}'

Dashboard (Traefik)

Traefik is the one chart here this repo doesn't install — k3s installs it itself. roles/k3s_traefik only adjusts it, through a HelmChartConfig merged over k3s's own HelmChart. Editing /var/lib/rancher/k3s/server/manifests/traefik.yaml on the node instead works until the next server restart rewrites it.

A stock k3s answers 404 on the dashboard, which looks like a broken install and isn't: Traefik still builds the dashboard, but the Traefik chart stopped creating the router that reaches it in v28. The role puts that router back on Traefik's internal traefik entrypoint, which isn't published on the Service — so it's reachable by port-forward and nothing else:

kubectl -n kube-system port-forward deploy/traefik 9000:9000
# then http://127.0.0.1:9000/dashboard/  — the trailing slash is required

Two things to check if that still 404s. curl -i http://127.0.0.1:9000/ping returning 200 means the port-forward is fine and only the router is missing (so the role hasn't run, or its HelmChartConfig hasn't reconciled yet — kubectl -n kube-system get ingressroute should list traefik-dashboard). And forward to deploy/traefik, not svc/traefik: the Service only publishes 80/443, so going through it lands you on the web entrypoint, where an unmatched request also returns 404.

Publishing it on a hostname is one commented line in inventory/group_vars/k3s_cluster.yml:

k3s_traefik_dashboard_host: traefik.turtlesystems.uk

That renders an IngressRoute on websecure with a cert-manager certificate and an Authentik forward-auth middleware in front. k3s_traefik_dashboard_auth_address must be set too — the role refuses to run otherwise rather than publishing an unauthenticated view of every router, service and middleware on the cluster. It's already set in that file, so in practice this is the single line above plus a DNS record pointing at Traefik's MetalLB address (kubectl -n kube-system get svc traefik).

On the Authentik side, first: a Proxy Provider in forward auth (domain level) mode, assigned to an application, added to the embedded outpost. Domain level rather than single-application because the latter needs /outpost.goauthentik.io/ routed to Authentik on the dashboard's own hostname, which from kube-system is a cross-namespace service reference Traefik rejects by default.

An IngressRoute rather than an Ingress, unlike every app here, because the dashboard is served by Traefik's internal api@internal and has no Kubernetes Service for an Ingress to point at. That's also why its certificate is an explicit Certificate resource — cert-manager watches Ingress, not IngressRoute.

Clearing the hostname again is a real teardown: the role deletes the manifest, and k3s's deploy controller garbage-collects the route, middleware and certificate it created.

The port-forward stays enabled alongside the published route on purpose. Once the dashboard is behind Authentik it depends on Authentik, which depends on CNPG, which depends on a healthy cluster — the things you'd open the dashboard to diagnose. Port-forward talks to the pod and traverses none of it, so it's the break-glass path, not a leftover.

Apps on the cluster

Apps go in k3s_apps: in inventory/group_vars/k3s_cluster.yml — the k3s equivalent of a host's stacks:/apps: list, in group_vars because an app is deployed to the cluster rather than to a node. Same entry shape as the other platforms (name, src, vault_path, optional db:, optional state:).

ansible-playbook playbooks/k3s.yml --tags apps
ansible-playbook playbooks/k3s.yml --tags apps -e only_apps=authentik

Ingress goes through the Traefik k3s already bundles, on the MetalLB address its Service holds (kubectl -n kube-system get svc traefik) — so adding an app costs a DNS record pointed at that one address, not a pool IP each.

Removal is state: absent on the entry, same tombstone rule as Unraid: keep the entry, don't delete it, or nothing is torn down and the workload keeps running unmanaged. Unlike Unraid it's a single pass — deleting the manifests is the teardown, because k3s's deploy controller garbage-collects what each file created — so src/<app>/ can go in the same commit. The database, PVCs and Vault path deliberately survive.

Authentik is currently the only app, and it got here by migrating off a hand-made Unraid container rather than being deployed fresh. If you're repeating that for something else, docs/authentik-migration.md is the worked procedure; the part worth knowing up front is that the deployed version must match the version a restored database was dumped from, because Authentik runs its migrations on startup and they don't go backwards.

SSH access

Ansible connects as ansible_user: root (set in build/config/ansible/inventory/hosts.yml) over SSH. Password auth would mean either an interactive prompt every run — which doesn't work from CI at all — or a plaintext password somewhere, so use a dedicated key pair instead:

ssh-keygen -t ed25519 -f ~/.ssh/unraid_ansible -C "ansible@unraid-iac" -N ""

Install the public half as an authorized key on every Unraid host this repo targets. Don't just ssh-copy-id it into ~/.ssh/authorized_keys — Unraid boots from the flash drive and /root lives on a RAM-backed overlay, so anything written there is gone on the next reboot. Persist it through /boot/config/ssh/root.authorized_keys instead, which Unraid copies into place at every boot:

cat ~/.ssh/unraid_ansible.pub | ssh root@<unraid-host> \
  'cat >> /boot/config/ssh/root.authorized_keys'

For manual/bootstrap runs, load the key into ssh-agent and Ansible picks it up automatically — no config file changes needed:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/unraid_ansible

For CI, rather than adding yet another place secrets live, the private key is stored in Vault (homelab/ci/ssh, see docs/vault-secrets.md) and the workflow fetches it at the start of each run:

vault kv put kv/homelab/ci/ssh PRIVATE_KEY=@~/.ssh/unraid_ansible

K3s Pis work the same way but as a non-root ansible user, not root, and with their own key pair (~/.ssh/k3s_ansible, Vault path homelab/ci/ssh-k3s) — see docs/vault-secrets.md for the full commands. Three differences from the Unraid steps above. The public key goes in the ordinary ~/.ssh/authorized_keys for the ansible user (Ubuntu's root filesystem isn't rebuilt from flash on every boot, so there's no persistence quirk to work around), and that user needs NOPASSWD sudo configured in /etc/sudoers.d/ — playbooks/k3s.yml installs k3s via become: true.

And the private half doesn't need loading into ssh-agent at all, even for a manual run: playbooks/k3s_ssh_key.yml fetches it from Vault and writes it to local/k3s/homelab-utils.key, and both playbooks/k3s.yml and playbooks/k3s_maintenance.yml import that as their first play. So a k3s run needs VAULT_ADDR/VAULT_TOKEN in the environment and nothing else — the same "Vault is the only place secrets live" arrangement CI already uses for the Unraid key, rather than a second setup step to remember. To fall back to a local copy (Vault down, say), skip the fetch and point at the key — overriding the path alone isn't enough, since the fetch play would still fail before anything else ran:

ansible-playbook playbooks/k3s.yml \
  -e k3s_ssh_key_fetch=false \
  -e ansible_ssh_private_key_file=~/.ssh/k3s_ansible

Bootstrapping (chicken-and-egg on the very first deploy)

CI/CD runs on Forgejo Actions — but Forgejo doesn't exist yet on the first run, and Actions needs a self-hosted runner besides. So the first deploy is manual:

1. Fill in build/config/ansible/inventory/hosts.yml with nas2's real address.
2. Set up the SSH key pair per "SSH access" above and install the public key on nas2 (and nas1, once you migrate anything to it).
3. vault kv put the secrets in docs/vault-secrets.md for homelab/shared/postgres, homelab/forgejo, and homelab/ci/ssh.
4. From WSL, with VAULT_ADDR / VAULT_TOKEN exported (vault login first) and the key loaded in ssh-agent:

   cd build/config/ansible
   ansible-playbook playbooks/unraid.yml --limit nas2
   

   This brings up shared Postgres, provisions the forgejo database/role, and brings up Forgejo. Use unraid.yml rather than deploy.yml at this point — no Proxmox guests exist yet, so the Proxmox play has nothing to target.
5. Log into Forgejo, create this repo there, push it.
6. Register a self-hosted Forgejo Actions runner reachable to Vault and to both Unraid hosts (labelled unraid-deploy — see .forgejo/workflows/deploy.yml). Give it an AppRole (VAULT_ROLE_ID / VAULT_SECRET_ID) scoped to read the kv/homelab/* paths it needs (including homelab/ci/ssh), stored as Forgejo Actions secrets alongside VAULT_ADDR.
7. From then on, pushes to main touching src/** or build/config/ansible/** deploy automatically — and only the stack(s) whose src/<stack>/ folder actually changed (see "Deploying one platform, one host, one stack" below); a change under build/config/ansible/ still triggers a full deploy of every stack on every host, since that's a change to how all of them get deployed.

Bringing Proxmox online later

The shared Postgres and its LXC are the first thing to stand up here, and the ordering is fiddly enough to have its own runbook: docs/postgres-proxmox.md. The short version, and the reason it isn't just "run Terraform":

Terraform stores its state in Postgres, so provisioning the Postgres LXC with Terraform would be circular. It's broken by keeping state on the CloudNativePG cluster on the k3s Pis instead — something this configuration has no hand in building — which means terraform apply now depends on the cluster being up. That's a cross-platform dependency the rest of the repo avoids, and it's worth it because the alternative is a local-state-then-migrate dance that has to be got right exactly once.

After that, docs/postgres-proxmox.md walks through the template download, terraform apply -target=module.postgres, checking the ZFS replication job actually exists, installing Postgres with ansible-playbook playbooks/proxmox.yml -e only_stacks=postgres, and playbooks/pve_host.yml for node backups. It also covers failing over between the two nodes and repointing clients afterwards.

Forgejo comes after all of that, on turtle-proxmox-02 — its database is the shared Postgres the runbook above creates, so none of it works until that one is finished. docs/forgejo-proxmox.md walks through the extra Vault key the native install needs, terraform apply -target=module.forgejo, ansible-playbook playbooks/proxmox.yml -e only_stacks=forgejo, and the parts Ansible deliberately does not do — moving repository data, and the DNS/SSH-port cutover.

Deploying one platform, one host, one stack

playbooks/deploy.yml runs everything; playbooks/unraid.yml, playbooks/proxmox.yml and playbooks/pve_host.yml do one slice each (Unraid stacks, Proxmox guests, and the Proxmox node's backup config respectively). Within any of them, --limit <host> narrows by host and -e only_stacks=<name>[,<name>...] narrows by stack or app (name = the name: in the host's stacks:/apps: list, e.g. forgejo or postgres — not the src: path):

# everything, both platforms
ansible-playbook playbooks/deploy.yml

# everything declared for nas2
ansible-playbook playbooks/unraid.yml --limit nas2

# just forgejo on nas2
ansible-playbook playbooks/unraid.yml --limit nas2 -e only_stacks=forgejo

# forgejo and its database, skipping any other stack on nas2
ansible-playbook playbooks/unraid.yml --limit nas2 -e only_stacks=forgejo,postgres

Note only_stacks doesn't resolve dependencies for you — if you limit to forgejo before postgres has ever been deployed, the DB provisioning step will fail because there's nothing to connect to. CI doesn't hit this: it computes only_stacks from which src/** paths actually changed in the push, so an unrelated app's push never touches Postgres or Forgejo.

deploy.yml runs Unraid before Proxmox by convention rather than necessity — the two platforms no longer depend on each other. Proxmox-side apps provision their databases against the Proxmox shared Postgres (192.168.50.54), and Terraform keeps its state on the CloudNativePG cluster on k3s; neither goes through Unraid. The node's backup config goes last, since a vzdump job pinned to specific VMIDs needs those guests to exist. Note that only_stacks has no meaning for pve_host.yml — it configures a node, not an app, and ignores it.

playbooks/k3s.yml is separate from all of the above — it isn't included in deploy.yml and doesn't take only_stacks, because a cluster service there isn't a stacks: entry to filter down to. It has two selectors of its own instead: --limit narrows which nodes, and --tags narrows which piece.

# the whole cluster: nodes, then the cluster services, then the kubeconfig
ansible-playbook playbooks/k3s.yml

# re-converge one worker, e.g. after reimaging it
ansible-playbook playbooks/k3s.yml --limit k3s-node-02

# just one cluster service, no node install across 4 Pis
ansible-playbook playbooks/k3s.yml --tags postgres

# every cluster service, still no node install
ansible-playbook playbooks/k3s.yml --tags services

The tags, one per play (see the playbook's header comment): nodes (both node plays), metallb, monitoring, postgres, cert-manager, traefik, services (all five of those together), apps, upgrades, kubeconfig. services and apps don't imply each other, so "everything except reinstalling k3s" is --tags services,apps. The SSH-key fetch play is tagged always rather than named, so it survives every filter — it's what any of the others connect with, not something you'd pick.

Deploying a single service this way skips the kubeconfig play too, so local/'s copy isn't refreshed. That only matters on a cluster rebuild; add --tags postgres,kubeconfig if you want both.

Worth knowing what a tagged run does and doesn't tell you: every service role just templates manifests into /var/lib/rancher/k3s/server/manifests/ for k3s's own controllers to reconcile, so the playbook finishing means the files landed, not that the workload is up. Watch that separately, e.g. kubectl -n shared-postgres get cluster,pods -w.

playbooks/k3s_maintenance.yml is separate again — see "Keeping it patched" above. It ignores --limit grouping in one sense worth knowing: serial: 1 still applies to whatever --limit narrows the run to, so limiting to two nodes still patches them one at a time, not together.

Adding a new app

1. src/<app>/common/vars.yml — env_defaults: for the config that doesn't depend on where it runs.

2. Build the platform side(s) you need:

   Unraid — src/<app>/ansible/unraid/ containing docker-compose.yml (static, ${VAR}-driven, joining unraid_shared with external: true if it needs the shared Postgres), vars.yml of overrides, and .env.example documenting every var with secrets blank. Optionally an icon.png and the net.unraid.docker.* labels, so the Docker page shows something other than a question mark.

   Proxmox — src/<app>/ansible/proxmox/ containing vars.yml, install.yml, and templates for the app's config file and systemd unit; plus src/<app>/terraform/ defining its LXC, called from build/config/terraform/main.tf.

3. Add its homelab/<app> path to Vault (docs/vault-secrets.md).

4. Declare it on the target host — stacks: in an Unraid host's host_vars/<host>.yml, or apps: in a Proxmox guest's. Same entry shape either way (name, src, vault_path, optional db: block).

5. Push — or run the relevant playbook manually before CI exists.

6. Leave the platform you didn't build as a README placeholder rather than deleting the folder.

Removing an app (Unraid)

Deleting the entry from stacks: does not remove anything. Ansible keeps no record of what it deployed last run, so an app that disappears from the list is simply never visited again — its containers keep running on the host, now unmanaged and invisible to the playbook. This is the one place the Unraid side doesn't behave like the Terraform side, where deleting a module call does destroy the LXC.

Removal is therefore an instruction, not an absence. Set state: absent on the entry and leave it in place:

stacks:
  - name: arr
    src: arr
    vault_path: homelab/arr
    state: absent

The next run of playbooks/unraid.yml runs docker compose down, then deletes the project folder (and with it the rendered .env). Because a host_vars/ edit is a change under build/config/ansible/, CI treats it as a full converge — so pushing that change is enough to action the teardown.

Keep the entry as a tombstone until you're sure. It's the only record that the app was deliberately removed rather than never deployed, and flipping state: back to present redeploys it.

What survives, and how to remove the rest

Bind mounts are untouched, so /mnt/user/appdata/<app> — which for a SQLite-backed app like Shelfarr is the application — survives a teardown and makes it reversible. External networks (caddy-net, unraid_shared) are left alone too; they belong to Unraid or to another stack. Both copies of the stack's icon are deleted without needing to be asked — the project-folder one goes with the folder, and the one under /mnt/user/appdata/icons/ is removed explicitly. Neither is state; they're copies of a file in the repo, and a redeploy puts them back.

Three things are destructive enough to stay opt-in, per entry:

Key	Effect
remove_volumes: true	Also delete the project's named volumes. Bind mounts are unaffected either way.
remove_images: local	Also delete its images (local or all, as docker compose down --rmi).
remove_database: true	Drop the Postgres database and its role. Only for stacks with a db: block.

Nothing removes the app's homelab/<app> path from Vault — that has its own lifecycle, and destroying it would make the teardown irreversible. Delete it by hand with vault kv metadata delete once you're finished with the app.

Order of operations

docker compose down reads the compose file to know what it's removing, so the teardown has to run before you delete src/<app>/ or the host's project folder. Delete those first and there's nothing left to tell Docker what belonged to the project — you're cleaning up by hand with docker rm and docker network rm instead. So: set state: absent, run the playbook, then delete files in a second pass.

There's no equivalent on the Proxmox side yet. lxc_app only ever installs, and a symmetric teardown needs a per-app uninstall.yml convention that no app implements — the role is still a skeleton. For now, removing a Proxmox app means stopping and disabling its systemd unit by hand, or destroying the container via Terraform.

Adding a host

Unraid: add it under unraid_servers in build/config/ansible/inventory/hosts.yml, then create build/config/ansible/inventory/host_vars/<name>.yml with its own stacks: list — each host only runs what it's assigned.

Proxmox: define the LXC as a module in src/<app>/terraform/, call it from build/config/terraform/main.tf, and terraform apply. The guest appears in inventory automatically via the API — no hosts.yml edit — but it still needs a host_vars/<guest>.yml declaring its apps: list.

K3s: add it as a new host under k3s_control_plane or k3s_workers (per inventory/hosts.yml → k3s_cluster) — no terraform apply, the Pi has to physically exist and have SSH access set up first (see "K3s (Raspberry Pi)" above). No host_vars/ entry needed beyond that: roles/k3s_node reads k3s_node_role off the group, not the host, so which group a new Pi joins is the only thing that decides whether it becomes another agent or a second control-plane node — and this repo's role doesn't support the latter (single server, no HA etcd) without changes to k3s_node/tasks/server.yml.