# Homelab IaC Deploy apps as code across two platforms: Unraid server(s), where apps run as Docker Compose stacks, and Proxmox, where Terraform creates an LXC and the app is installed into it natively. Ansible is the deployment tool on both. Secrets are pulled from HashiCorp Vault at deploy time; CI/CD is Forgejo Actions once Forgejo itself is running. There's also a third piece of infrastructure alongside those two: a bare-metal k3s cluster (`homelab-utils`) across 4 Raspberry Pis, bootstrapped by Ansible too. It doesn't fit the `src//` table above the way Unraid/Proxmox apps do — Terraform never provisions it, and no `stacks:`/`apps:` list drives what's on it — so it gets its own inventory group and role instead. It does run shared services deployed via Helm, though: `k3s_monitoring`, `k3s_metallb`, and the cluster's own shared Postgres (`k3s_postgres`, a third platform for `src/shared/postgres/`). See `playbooks/k3s.yml` and "K3s (Raspberry Pi)" below. See `CLAUDE.md` for the short version of the architecture decisions. ## Layout ``` src/ / common/vars.yml config that holds on either platform ansible/ unraid/ docker-compose.yml + .env.example + platform overrides (+ optional icon.png for the Unraid Docker page) proxmox/ install.yml + config/systemd templates + platform overrides kubernetes/ vars.yml + *.yaml.j2 manifests (normally a HelmChart CR) terraform/ this app's LXC definition (Proxmox only) shared// same shape, for services multiple apps use build/config/ ansible/ inventory/ hosts.yml static: unraid_servers, proxmox_nodes, k3s_cluster proxmox.yml dynamic: proxmox_guests, from the Proxmox API group_vars/all.yml Vault connection, Compose Manager project path group_vars/k3s_*.yml k3s cluster name/version, per-role vars (server/agent) host_vars/.yml which stacks/apps deploy there (k3s apps are in group_vars/k3s_cluster.yml instead) roles/compose_stack/ Unraid: sync compose + render .env + provision DB + up (deploy.yml / remove.yml, picked by the entry's state:) roles/lxc_app/ Proxmox: install natively + render config + systemd roles/k3s_app/ k3s: render app manifests + Secret + provision DB (deploy.yml / remove.yml, picked by the entry's state:) roles/pve_backup/ Proxmox node: NAS backup storage + vzdump schedule roles/k3s_node/ k3s Pi: cgroups/swap prep + install server or agent roles/k3s_monitoring/ k3s control plane: lean kube-prometheus-stack via HelmChart CR roles/k3s_metallb/ k3s control plane: MetalLB (LoadBalancer IPs) via HelmChart CR roles/k3s_postgres/ k3s control plane: shared Postgres (CloudNativePG) via HelmChart CR roles/k3s_cert_manager/ k3s control plane: cert-manager + ClusterIssuer via HelmChart CR roles/unattended_upgrades/ any apt host: hands-off patching, no auto-reboot roles/k3s_maintenance/ k3s Pi: drain + reboot-if-required + uncordon, one at a time playbooks/ deploy.yml everything (Unraid + Proxmox — not k3s, see below) unraid.yml just Unraid proxmox.yml just the Proxmox guests pve_host.yml just the Proxmox node (backups) k3s.yml bootstraps the k3s cluster (manual-only, not in deploy.yml) k3s_maintenance.yml rolls pending reboots across it, serial: 1 (manual-only) terraform/ Proxmox provider, pg state backend, LXC module calls .forgejo/workflows/ CI/CD (needs a self-hosted runner, see below) docs/vault-secrets.md Vault path/key reference docs/authentik-migration.md hand-run cutover: Authentik from Unraid to k3s ``` Terraform's scope stops at the guest — it creates the LXC and nothing more, because the Proxmox provider has no way to install an application. Ansible picks up from there, which is why it's the common tool and Terraform is Proxmox-only. ## How config is layered An app's settings are split by whether they survive a change of platform: - `src//common/vars.yml` — version, ports, domain, database name/user. - `src//ansible//vars.yml` — the rest. Data paths differ (`/mnt/user/appdata/forgejo` vs `/var/lib/forgejo`), and so does how the database is reached (the `shared-postgres` Docker network alias only exists on Unraid). Both roles merge them the same way: common, then platform overrides, then Vault secrets last. Nothing is restated between platforms. ## How a deploy works **Unraid.** For each stack in a host's `stacks:` list, `compose_stack`: 1. Copies `src//ansible/unraid/docker-compose.yml` to the host's Compose Manager project folder, unmodified — it's static and only references `${VAR}`. 2. Merges the two `vars.yml` layers with the stack's Vault KV path and renders the project's `.env`. Nothing secret ever touches git. 3. If the stack declares a `db:` block, ensures its database/role exist on the shared Postgres instance (idempotent — safe to add new apps to the shared service over time without touching existing data). 4. Copies `icon.png`, if the stack ships one, to both the project folder and `/mnt/user/appdata/icons/` (see "Icons on the Unraid Docker page" below). 5. Runs `docker compose up -d` for the stack. **Proxmox.** `terraform apply` creates the LXC first. Then, for each app in a guest's `apps:` list, `lxc_app` does the same config merge and database provisioning, runs that app's `ansible/proxmox/install.yml`, and manages its systemd unit instead of a Compose project. **k3s.** For each app in `k3s_apps:` (`group_vars/k3s_cluster.yml`, not host_vars — an app belongs to the cluster, not a node), `k3s_app`: 1. Does the same two-layer config merge, using `src//ansible/kubernetes/vars.yml` as the platform layer. 2. Provisions the app's database/role on the CloudNativePG cluster if it declares a `db:` block — the same `community.postgresql` tasks the other two platforms use, pointed at MetalLB's LoadBalancer address for the CNPG primary. 3. Renders the app's Vault path into a Kubernetes `Secret` (plus the `Namespace`), keys passed through verbatim. 4. Renders every `*.yaml.j2` the app ships into `/var/lib/rancher/k3s/server/manifests/`, where k3s's own deploy and helm controllers pick them up. Nothing here talks to the Kubernetes API — no `helm`, no kubeconfig, no `kubernetes.core`. Ansible writes files; k3s reconciles them. The consequence worth remembering: **a green run means the manifests landed, not that the workload came up.** Check with `kubectl -n get pods`. Secrets land on disk in the rendered `.env` (root-readable, on the Unraid box) or, on k3s, in a 0600 manifest and then in etcd as an ordinary Kubernetes Secret — this is "fetch from Vault at deploy time," not a zero-secrets-at-rest model. Fine for a homelab; revisit if that changes. ## Icons on the Unraid Docker page There are two icons per stack, and they come from completely different places. Commit **one** `src//ansible/unraid/icon.png` and the deploy feeds both. A stack running more than one container can commit `icon-.png` alongside it for the extra services — see "More than one container" below. ### The stack row Compose Manager draws the collapsible header row for the whole project, and serves its icon straight off disk from `icon.png` in the project folder — `/boot/config/plugins/compose.manager/projects//icon.png`, the same directory the compose file and rendered `.env` land in. No label, no template, no URL: the file is there or the row has no icon. `.jpg`, `.gif`, `.svg` and an extensionless `icon` also work, since the plugin serves the file itself rather than handing a path to the webgui — but stick to PNG so the same file can serve the labels below. This is a feature of **Compose Manager Plus** (`mstrhakr/compose_plugin`), the maintained fork. The original Docker Compose Manager plugin (`dcflachs`, archived April 2026) has no icon support at all — on that one the copied file is simply ignored, so the deploy is safe either way, you just won't see it. ### The containers under it Unraid's Docker page gets each *container's* icon, WebUI link and console shell out of the dockerMan template that created it, under `/boot/config/plugins/dockerMan/templates-user/`. A Compose stack never creates one, which is why compose-deployed containers show a question mark and no WebUI entry. Unraid 6.10+ falls back to Docker labels when there's no template, so each service sets them itself: ```yaml labels: net.unraid.docker.icon: ${STACK_ICON} net.unraid.docker.webui: "http://[IP]:${FORGEJO_HTTP_PORT}/" net.unraid.docker.shell: sh ``` They're per **service**, not per stack — a stack with two containers labels each one separately, and can point them at different icons. - **icon** — `${STACK_ICON}` is set by the role, not by hand. The same committed `icon.png` is copied a second time to `/mnt/user/appdata/icons/.png` and the label points there; ship no `icon.png` and the variable renders empty, leaving the placeholder. To use a hosted image instead, set `STACK_ICON` in the app's `ansible/unraid/vars.yml` — it's merged over the role's value. A second copy rather than reusing the project-folder one because this label is a path the *webgui* resolves on every Docker page load, and that path shouldn't run through the flash drive. **PNG only here** — unlike the stack row, this goes through the webgui's own icon handling, where SVG renders as the question-mark fallback and WebP renders nothing at all. A remote URL is fetched when the page renders, so an unreachable host breaks the icon — the reason a committed file is the default. - **webui** — `[IP]` is substituted with the host's address. The port must be the *published* one, so interpolate the stack's own port variable rather than the container port. Omit the label entirely for something with no web interface (as `shared/postgres` does) — an empty value still draws a WebUI entry that goes nowhere. - **shell** — `sh` or `bash`, whichever the image actually has. Alpine-based images (Forgejo) need `sh`; Debian-based ones (Postgres, Shelfarr) can take `bash`. One caveat: editing a compose container's labels through the Unraid UI writes a template for it, and the template then wins over the labels until you clear it. Change these in the compose file and redeploy, not in the webgui. ### More than one container `${STACK_ICON}` is a single value — the stack's `icon.png` — so a stack with two services would otherwise have to give both the same image. Since the labels are per-service, commit `icon-.png` next to `icon.png` for each additional container, named for the **compose service**: ``` src/arr/ansible/unraid/ icon.png → ${STACK_ICON} → /mnt/user/appdata/icons/arr.png icon-prowlarr.png → ${STACK_ICON_PROWLARR} → /mnt/user/appdata/icons/arr-prowlarr.png ``` The variable is the service name uppercased with `-` folded to `_` (`icon-shelfarr-libation.png` → `${STACK_ICON_SHELFARR_LIBATION}`). These get the appdata copy only — there is one Compose Manager stack row and `icon.png` already has it — and, like `STACK_ICON`, each can be overridden with a hosted URL by setting the same name in the app's `ansible/unraid/vars.yml`. The variable only exists when the file does. Interpolating `${STACK_ICON_FOO}` with no `icon-foo.png` committed gets Compose's unset-variable warning and the question-mark placeholder — so if the icon is meant to be a URL, set it in `vars.yml` rather than relying on the file. ## Backups (Proxmox) Proxmox guests keep **all** their state on their own rootfs. Nothing is bind-mounted in from the NAS, which is the opposite of the obvious instinct — the reason is that `vzdump` deliberately excludes bind mounts, so a bind-mounted repository directory would be the one thing missing from the backup that was made to protect it. Keeping everything local means one archive is a complete copy of the guest's filesystem. `playbooks/pve_host.yml` (role: `pve_backup`) configures the off-box copy on the node: an NFS storage pointing at the NAS, and a scheduled `vzdump` job writing to it. Retention is the storage's `prune-backups` settings. Point it at a Proxmox Backup Server instead by setting `pve_backup_storage_type: pbs` in `host_vars/.yml`. **Data that lives outside a container is that app's problem, not vzdump's.** Forgejo's database is on the shared Postgres over on Unraid, so restoring its container alone would give back every repository with no issues, pull requests, users or permissions. The fix is a `pg_dump` on a systemd timer *inside* the container, writing into its own filesystem — which puts the dump inside the same archive as the repositories it belongs to. That only works if it finishes before the backup window, so the two schedules are a pair: | | Set in | Default | |---|---|---| | Forgejo database dump | `src/forgejo/ansible/proxmox/vars.yml` → `FORGEJO_DB_DUMP_ONCALENDAR` | 01:30 | | vzdump job | `inventory/host_vars/pve.yml` → `pve_backup_schedule` | 02:00 | Any future app with external state should follow the same pattern. Restore steps are in `src/forgejo/ansible/proxmox/README.md`. None of this covers the Unraid side, which has its own backup arrangements outside this repo. ## Prerequisites ### Both platforms - Ansible control node needs to be Linux — **on Windows, run it from WSL2**, not natively. - If this repo is checked out on a Windows mount (e.g. under `/mnt/c/...`, which is where a Seafile-synced folder ends up) rather than the native WSL filesystem, Ansible will warn that `ansible.cfg` is being ignored because the directory looks world-writable — that's DrvFs reporting `777` on everything, not a real permissions problem. Either `export ANSIBLE_CONFIG="$(pwd)/ansible.cfg"` before running (from the `build/config/ansible/` dir), or fix it permanently by adding to `/etc/wsl.conf` inside WSL: ```ini [automount] options = "metadata,umask=22,fmask=11" ``` then `wsl --shutdown` from PowerShell and reopen the shell. - `pip install ansible hvac psycopg2-binary` - `ansible-galaxy collection install -r build/config/ansible/requirements.yml` - A reachable Vault instance, with secrets populated per `docs/vault-secrets.md`. ### Unraid - Root SSH enabled on the Unraid box(es) (Settings → Management Access). - Compose Manager (Docker Compose plugin) installed on Unraid — already done. - **Python3 on each Unraid host itself**, not just the controller: the `compose_stack` role's tasks (`file`, `copy`, `template`, `docker_compose_v2`) run on the target, and Ansible modules — unlike `raw`/ad hoc shell — need a Python interpreter there to execute at all. Stock Unraid doesn't ship one. Installing it by hand into the running system doesn't survive a reboot, same reason as the SSH-key gotcha above — Unraid rebuilds its root filesystem from the flash drive on every boot. Install the **NerdTools** plugin (Community Applications → search "NerdTools", formerly "NerdPack") and enable `python3` in its package list — it reinstalls whatever you've selected on every boot itself, same mechanism Compose Manager already relies on. `ansible.cfg` already sets `interpreter_python = auto_silent`, so Ansible finds it automatically wherever NerdTools puts it; no path to hardcode. Verify after installing: ```sh ansible unraid_servers -m ping ``` ### Proxmox - Terraform CLI (>= 1.6) on whatever runs `terraform apply`. - A Proxmox API token with rights to create containers, stored in Vault at `homelab/ci/proxmox` (see `docs/vault-secrets.md`). Both the Terraform provider and Ansible's dynamic inventory authenticate with it. - A `terraform_state` database and `terraform` role on the **CloudNativePG cluster on k3s** (192.168.50.81), for `backend "pg"` — not on either shared Postgres this repo deploys, which would be circular. See "Bootstrapping" below. - Container templates downloaded on the Proxmox node for whatever OS the LXCs are built from. - An NFS export on the NAS for backup archives, reachable from the Proxmox node — set it in `inventory/host_vars/pve.yml`. See "Backups (Proxmox)". - Root SSH from the Ansible controller to the Proxmox node itself, not just the guests: `playbooks/pve_host.yml` configures the node over SSH. - Unlike Unraid, nothing special is needed *inside* the guests: they're ordinary Linux containers with Python already present, so Ansible works without the NerdTools workaround above. ### K3s (Raspberry Pi) - 4 Raspberry Pis running Ubuntu Server (64-bit), already imaged and on the network at the addresses in `inventory/hosts.yml` → `k3s_cluster`. - An `ansible` user on each Pi with **NOPASSWD sudo** and this repo's k3s SSH key installed as an authorized key — see "SSH access" below. Not automated by this repo; create it by hand (or via cloud-init at image time) before the first run of `playbooks/k3s.yml`. - `homelab/k3s-homelab-utils` → `K3S_TOKEN` set in Vault *before* the first run — see `docs/vault-secrets.md`. Both the server and every agent read this same fixed value rather than one generating it and handing it to the other, which is what makes a rebuild reproducible. - Nothing else special: `python3` ships with Ubuntu Server, and the k3s install script (`get.k3s.io`) handles containerd, the systemd unit, and everything else that isn't Pi-specific. The one Pi-specific thing — ensuring the memory cgroup controller is on — is handled by `roles/k3s_node` itself, not a prerequisite here. #### Keeping it patched `playbooks/k3s.yml` also applies `roles/unattended_upgrades` to every node — apt updates install themselves on their own daily schedule, no login required. It sets `Unattended-Upgrade::Automatic-Reboot "false"` though, so a kernel or containerd update that needs a reboot to take effect just sits applied-but-inactive until one happens; blindly auto-rebooting a k3s node takes its pods down with no warning. `playbooks/k3s_maintenance.yml` is the other half: it checks `/var/run/reboot-required` on each node and, only where it's set, cordons and drains the node, reboots it, waits for it to report `Ready` again, then uncordons it — one node at a time (`serial: 1`), so patching the cluster never means dropping every workload at once. Manual-only for now, same as `playbooks/k3s.yml`: ```sh ansible-playbook playbooks/k3s_maintenance.yml ``` Eventually this is meant to run on a schedule from a self-hosted Forgejo Actions runner rather than by hand — not wired up yet, see the header comment in the playbook. #### Metrics (Prometheus) `playbooks/k3s.yml` also deploys `roles/k3s_monitoring`: a lean kube-prometheus-stack (Prometheus + prometheus-operator + node-exporter + kube-state-metrics — no Grafana, no Alertmanager) so tools like OpenLens can show node/pod metrics. It's installed as a `HelmChart` custom resource dropped into k3s's own auto-deploying manifests directory (`/var/lib/rancher/k3s/server/manifests/`) rather than run through a `helm` binary — k3s's bundled helm-controller reconciles it the same way it installs its own Traefik and ServiceLB, so this needs no extra Ansible collection or kubeconfig on the controller. See `roles/k3s_monitoring/defaults/main.yml` for the resource sizing (tuned for a Raspberry Pi 4, not a real node) and why the control-plane component scrapers (`kubeControllerManager`, `kubeScheduler`, `kubeProxy`, `kubeEtcd`) are disabled — k3s doesn't expose those the way kube-prometheus-stack expects, so leaving them on just produces permanently-"down" targets. To point OpenLens at it: open the cluster's Settings → Metrics, set Prometheus to "Auto detect" or explicitly to the `Operator` provider — it should find the `kube-prometheus-stack-prometheus` service in the `monitoring` namespace via the API server proxy, the same path OpenLens already uses to reach the cluster, so nothing needs to be exposed outside it. #### LoadBalancer IPs (MetalLB) `playbooks/k3s.yml` also deploys `roles/k3s_metallb`: MetalLB in L2 mode, handing out real LAN IPs to `type: LoadBalancer` Services instead of the `ClusterIP`-only world k3s would otherwise leave homelab-utils apps in. It's installed the same way as monitoring — a `HelmChart` CR for k3s's helm-controller to reconcile — plus a plain `IPAddressPool`/`L2Advertisement` manifest dropped in the same directory; k3s's deploy controller applies both kinds of file and retries the config manifest until the HelmChart's CRDs exist. MetalLB replaces k3s's bundled ServiceLB (Klipper), it doesn't run alongside it — both would otherwise compete to satisfy the same Services. That's why `inventory/group_vars/k3s_cluster.yml` sets `k3s_server_extra_args: ['--disable=servicelb']` — a separate, server-only var from `k3s_extra_args`, since `k3s agent` doesn't understand `--disable` and would fail to start if it were passed there too. The IP pool itself is `k3s_metallb_address_range` in the same file — a range on the cluster's LAN (`192.168.50.0/24`) that DHCP and every static assignment in `inventory/hosts.yml` steer clear of. There's no built-in default: the role fails fast if it's still empty. Enabling `--disable=servicelb` on an already-running cluster doesn't need a separate step: `roles/k3s_node` notices its exec line has changed and reinstalls (restarting just the `k3s` service, not the node) the next time `playbooks/k3s.yml` runs — see `CLAUDE.md` → "Key decisions". #### Postgres (CloudNativePG) `playbooks/k3s.yml` also deploys `roles/k3s_postgres`: CloudNativePG (CNPG), a Postgres operator, installed the same way as monitoring and MetalLB — a `HelmChart` CR for k3s's helm-controller to reconcile — plus a plain `Cluster` CR (the actual database) and a `Secret` (superuser credentials), dropped in the same auto-deploying manifests directory. This is a third platform for `src/shared/postgres/`, alongside the Unraid Compose stack and the Proxmox placeholder: `roles/k3s_postgres` layers `src/shared/postgres/common/vars.yml` with `src/shared/postgres/ansible/kubernetes/vars.yml` the same way `compose_stack`/`lxc_app` layer an app's config, and pulls the superuser password from the same `homelab/shared/postgres` Vault path those platforms already use (see `docs/vault-secrets.md`) — a separate physical instance, not a shared login across platforms, just the same identity. 1 primary + 1 replica, kept off the control-plane Pi via `nodeAffinity` (see `roles/k3s_postgres/templates/postgres-cluster.yaml.j2`) — CNPG's own pod anti-affinity then spreads the two across the 3 worker Pis. Storage is k3s's default `local-path` StorageClass; that's node-local, but resilience here comes from CNPG's own streaming replication between instances, not from shared storage, so losing one instance's disk doesn't lose the data as long as the other instance is up. HA replication only for now — no `ScheduledBackup` to the NAS or anywhere else yet. To connect: `kubectl -n shared-postgres get svc` lists four Services. Three are CNPG's own, all `ClusterIP` — `shared-postgres-rw` (the current primary), `-ro` (replicas only) and `-r` (any instance). An app running on the cluster uses the first of those by DNS and needs nothing else: ``` shared-postgres-rw.shared-postgres.svc.cluster.local:5432 ``` The fourth, `shared-postgres-lb`, is a `LoadBalancer` on `k3s_postgres_loadbalancer_ip` (`inventory/group_vars/k3s_cluster.yml` — inside the MetalLB pool above), which is how the cluster is reached from outside it: ```sh psql -h 192.168.50.81 -U postgres # password: homelab/shared/postgres ``` It's declared in the `Cluster` CR as a CNPG *managed service* rather than as a Service manifest of our own, so the operator keeps its selector pointed at whichever instance is currently primary — a hand-written Service would need re-pointing by hand after a failover. Credentials are the `shared-postgres-superuser-vault` Secret Ansible renders; CNPG separately auto-creates `shared-postgres-app` for the default `app` database it bootstraps, which nothing uses yet. The `-vault` suffix keeps that Secret clear of `-superuser`, the name CNPG uses for the superuser secret it generates itself. Name our own object that and the operator treats it as one it already authored, so a rotated password reaches the Secret and never reaches the database — the Secret reads correctly, `psql` from the LAN keeps failing authentication, and only `select rolpassword is null from pg_authid where rolname='postgres'` inside the pod shows the disagreement. That LoadBalancer is what makes the controller-reachable `host:port` the `community.postgresql` tasks need — the same ones `compose_stack` and `lxc_app` use to create an app's database and role — available here too, and `roles/k3s_app` now uses exactly those tasks for any app declaring a `db:` block. Note the deliberate asymmetry, which reads like a mistake until you know why: provisioning uses `192.168.50.81` (the LoadBalancer) because it runs on the Ansible controller, which is off-cluster and can't route to a ClusterIP; apps use `shared-postgres-rw.shared-postgres.svc.cluster.local` because sending pod traffic out to the LAN and back would put MetalLB's L2 speaker in the path of every query for nothing. Two addresses, one database, each correct for its caller. Without `kubectl` to hand, or if MetalLB is having a bad day, the in-cluster routes still work: ```sh kubectl -n shared-postgres port-forward svc/shared-postgres-rw 5432:5432 kubectl -n shared-postgres exec -it shared-postgres-1 -- psql -U postgres ``` **No backups.** HA replication only — a streaming replica on another Pi, which covers a dead SD card and not a dropped table, a bad migration, or a mistyped `DROP`. This mattered less when the cluster held nothing; it matters now that Authentik's database lives here rather than on Unraid. A CNPG `ScheduledBackup` to the NAS is the obvious next piece of work and isn't built yet. #### TLS (cert-manager) `playbooks/k3s.yml` also applies `roles/k3s_cert_manager`: cert-manager plus one `ClusterIssuer` named `letsencrypt`, both via the usual `HelmChart` CR + plain manifest pair. An app's Ingress opts in with a single annotation — ```yaml annotations: cert-manager.io/cluster-issuer: letsencrypt ``` — and cert-manager creates and renews the certificate on its own. Two values have no default and the role refuses to run without them, both in `inventory/group_vars/k3s_cluster.yml`: - `k3s_cert_manager_acme_email` — an ACME account is registered against it. - `k3s_cert_manager_solver` — the DNS-01 stanza for your DNS provider, rendered into the issuer as-is. A commented Cloudflare example is in that file; for anything else take the stanza from [cert-manager's docs](https://cert-manager.io/docs/configuration/acme/dns01/). Its API token goes in Vault at `homelab/k3s-cert-manager`. DNS-01 rather than HTTP-01 because HTTP-01 needs Let's Encrypt to reach this cluster from the internet on port 80, which it can't. That has a useful consequence: a certificate can be issued **before** DNS points at the cluster, so a migration's TLS is settled before its cutover rather than after. While working out a solver, point `k3s_cert_manager_acme_server` at Let's Encrypt staging — production allows five failed validations per hostname per hour, and exhausting it means waiting, not retrying. Check it registered: ```sh kubectl get clusterissuer letsencrypt -o jsonpath='{.status.conditions[*].message}' ``` #### Dashboard (Traefik) Traefik is the one chart here this repo doesn't install — k3s installs it itself. `roles/k3s_traefik` only adjusts it, through a `HelmChartConfig` merged over k3s's own `HelmChart`. Editing `/var/lib/rancher/k3s/server/manifests/traefik.yaml` on the node instead works until the next server restart rewrites it. A stock k3s answers **404** on the dashboard, which looks like a broken install and isn't: Traefik still builds the dashboard, but the Traefik chart stopped creating the router that reaches it in v28. The role puts that router back on Traefik's internal `traefik` entrypoint, which isn't published on the Service — so it's reachable by port-forward and nothing else: ```sh kubectl -n kube-system port-forward deploy/traefik 9000:9000 # then http://127.0.0.1:9000/dashboard/ — the trailing slash is required ``` Two things to check if that still 404s. `curl -i http://127.0.0.1:9000/ping` returning 200 means the port-forward is fine and only the router is missing (so the role hasn't run, or its `HelmChartConfig` hasn't reconciled yet — `kubectl -n kube-system get ingressroute` should list `traefik-dashboard`). And forward to `deploy/traefik`, not `svc/traefik`: the Service only publishes 80/443, so going through it lands you on the `web` entrypoint, where an unmatched request also returns 404. **Publishing it on a hostname** is one commented line in `inventory/group_vars/k3s_cluster.yml`: ```yaml k3s_traefik_dashboard_host: traefik.turtlesystems.uk ``` That renders an `IngressRoute` on `websecure` with a cert-manager certificate and an Authentik forward-auth middleware in front. `k3s_traefik_dashboard_auth_address` must be set too — the role refuses to run otherwise rather than publishing an unauthenticated view of every router, service and middleware on the cluster. It's already set in that file, so in practice this is the single line above plus a DNS record pointing at Traefik's MetalLB address (`kubectl -n kube-system get svc traefik`). On the Authentik side, first: a Proxy Provider in **forward auth (domain level)** mode, assigned to an application, added to the embedded outpost. Domain level rather than single-application because the latter needs `/outpost.goauthentik.io/` routed to Authentik on the dashboard's own hostname, which from `kube-system` is a cross-namespace service reference Traefik rejects by default. An `IngressRoute` rather than an `Ingress`, unlike every app here, because the dashboard is served by Traefik's internal `api@internal` and has no Kubernetes Service for an `Ingress` to point at. That's also why its certificate is an explicit `Certificate` resource — cert-manager watches `Ingress`, not `IngressRoute`. Clearing the hostname again is a real teardown: the role deletes the manifest, and k3s's deploy controller garbage-collects the route, middleware and certificate it created. The port-forward stays enabled alongside the published route on purpose. Once the dashboard is behind Authentik it depends on Authentik, which depends on CNPG, which depends on a healthy cluster — the things you'd open the dashboard to diagnose. Port-forward talks to the pod and traverses none of it, so it's the break-glass path, not a leftover. #### Apps on the cluster Apps go in `k3s_apps:` in `inventory/group_vars/k3s_cluster.yml` — the k3s equivalent of a host's `stacks:`/`apps:` list, in group_vars because an app is deployed to the cluster rather than to a node. Same entry shape as the other platforms (name, src, vault_path, optional `db:`, optional `state:`). ```sh ansible-playbook playbooks/k3s.yml --tags apps ansible-playbook playbooks/k3s.yml --tags apps -e only_apps=authentik ``` Ingress goes through the Traefik k3s already bundles, on the MetalLB address its Service holds (`kubectl -n kube-system get svc traefik`) — so adding an app costs a DNS record pointed at that one address, not a pool IP each. Removal is `state: absent` on the entry, same tombstone rule as Unraid: keep the entry, don't delete it, or nothing is torn down and the workload keeps running unmanaged. Unlike Unraid it's a single pass — deleting the manifests *is* the teardown, because k3s's deploy controller garbage-collects what each file created — so `src//` can go in the same commit. The database, PVCs and Vault path deliberately survive. **Authentik** is currently the only app, and it got here by migrating off a hand-made Unraid container rather than being deployed fresh. If you're repeating that for something else, `docs/authentik-migration.md` is the worked procedure; the part worth knowing up front is that the deployed version must match the version a restored database was dumped from, because Authentik runs its migrations on startup and they don't go backwards. ## SSH access Ansible connects as `ansible_user: root` (set in `build/config/ansible/inventory/hosts.yml`) over SSH. Password auth would mean either an interactive prompt every run — which doesn't work from CI at all — or a plaintext password somewhere, so use a dedicated key pair instead: ```sh ssh-keygen -t ed25519 -f ~/.ssh/unraid_ansible -C "ansible@unraid-iac" -N "" ``` Install the public half as an authorized key on **every** Unraid host this repo targets. Don't just `ssh-copy-id` it into `~/.ssh/authorized_keys` — Unraid boots from the flash drive and `/root` lives on a RAM-backed overlay, so anything written there is gone on the next reboot. Persist it through `/boot/config/ssh/root.authorized_keys` instead, which Unraid copies into place at every boot: ```sh cat ~/.ssh/unraid_ansible.pub | ssh root@ \ 'cat >> /boot/config/ssh/root.authorized_keys' ``` For manual/bootstrap runs, load the key into `ssh-agent` and Ansible picks it up automatically — no config file changes needed: ```sh eval "$(ssh-agent -s)" ssh-add ~/.ssh/unraid_ansible ``` For CI, rather than adding yet another place secrets live, the private key is stored in Vault (`homelab/ci/ssh`, see `docs/vault-secrets.md`) and the workflow fetches it at the start of each run: ```sh vault kv put kv/homelab/ci/ssh PRIVATE_KEY=@~/.ssh/unraid_ansible ``` **K3s Pis** work the same way but as a non-root `ansible` user, not root, and with their own key pair (`~/.ssh/k3s_ansible`, Vault path `homelab/ci/ssh-k3s`) — see `docs/vault-secrets.md` for the full commands. Three differences from the Unraid steps above. The public key goes in the ordinary `~/.ssh/authorized_keys` for the `ansible` user (Ubuntu's root filesystem isn't rebuilt from flash on every boot, so there's no persistence quirk to work around), and that user needs NOPASSWD sudo configured in `/etc/sudoers.d/` — `playbooks/k3s.yml` installs k3s via `become: true`. And the private half doesn't need loading into `ssh-agent` at all, even for a manual run: `playbooks/k3s_ssh_key.yml` fetches it from Vault and writes it to `local/k3s/homelab-utils.key`, and both `playbooks/k3s.yml` and `playbooks/k3s_maintenance.yml` import that as their first play. So a k3s run needs `VAULT_ADDR`/`VAULT_TOKEN` in the environment and nothing else — the same "Vault is the only place secrets live" arrangement CI already uses for the Unraid key, rather than a second setup step to remember. To fall back to a local copy (Vault down, say), skip the fetch **and** point at the key — overriding the path alone isn't enough, since the fetch play would still fail before anything else ran: ```sh ansible-playbook playbooks/k3s.yml \ -e k3s_ssh_key_fetch=false \ -e ansible_ssh_private_key_file=~/.ssh/k3s_ansible ``` ## Bootstrapping (chicken-and-egg on the very first deploy) CI/CD runs on Forgejo Actions — but Forgejo doesn't exist yet on the first run, and Actions needs a self-hosted runner besides. So the first deploy is manual: 1. Fill in `build/config/ansible/inventory/hosts.yml` with nas2's real address. 2. Set up the SSH key pair per "SSH access" above and install the public key on nas2 (and nas1, once you migrate anything to it). 3. `vault kv put` the secrets in `docs/vault-secrets.md` for `homelab/shared/postgres`, `homelab/forgejo`, and `homelab/ci/ssh`. 4. From WSL, with `VAULT_ADDR` / `VAULT_TOKEN` exported (`vault login` first) and the key loaded in `ssh-agent`: ```sh cd build/config/ansible ansible-playbook playbooks/unraid.yml --limit nas2 ``` This brings up shared Postgres, provisions the `forgejo` database/role, and brings up Forgejo. Use `unraid.yml` rather than `deploy.yml` at this point — no Proxmox guests exist yet, so the Proxmox play has nothing to target. 5. Log into Forgejo, create this repo there, push it. 6. Register a self-hosted Forgejo Actions runner reachable to Vault and to both Unraid hosts (labelled `unraid-deploy` — see `.forgejo/workflows/deploy.yml`). Give it an AppRole (`VAULT_ROLE_ID` / `VAULT_SECRET_ID`) scoped to read the `kv/homelab/*` paths it needs (including `homelab/ci/ssh`), stored as Forgejo Actions secrets alongside `VAULT_ADDR`. 7. From then on, pushes to `main` touching `src/**` or `build/config/ansible/**` deploy automatically — and only the stack(s) whose `src//` folder actually changed (see "Deploying one platform, one host, one stack" below); a change under `build/config/ansible/` still triggers a full deploy of every stack on every host, since that's a change to how *all* of them get deployed. ### Bringing Proxmox online later The shared Postgres and its LXC are the first thing to stand up here, and the ordering is fiddly enough to have its own runbook: **`docs/postgres-proxmox.md`**. The short version, and the reason it isn't just "run Terraform": Terraform stores its state in Postgres, so provisioning the Postgres LXC with Terraform would be circular. It's broken by keeping state on the CloudNativePG cluster on the k3s Pis instead — something this configuration has no hand in building — which means `terraform apply` now depends on the cluster being up. That's a cross-platform dependency the rest of the repo avoids, and it's worth it because the alternative is a local-state-then-migrate dance that has to be got right exactly once. After that, `docs/postgres-proxmox.md` walks through the template download, `terraform apply -target=module.postgres`, checking the ZFS replication job actually exists, installing Postgres with `ansible-playbook playbooks/proxmox.yml -e only_stacks=postgres`, and `playbooks/pve_host.yml` for node backups. It also covers failing over between the two nodes and repointing clients afterwards. Forgejo comes after all of that, on **turtle-proxmox-02** — its database is the shared Postgres the runbook above creates, so none of it works until that one is finished. `docs/forgejo-proxmox.md` walks through the extra Vault key the native install needs, `terraform apply -target=module.forgejo`, `ansible-playbook playbooks/proxmox.yml -e only_stacks=forgejo`, and the parts Ansible deliberately does not do — moving repository data, and the DNS/SSH-port cutover. ## Deploying one platform, one host, one stack `playbooks/deploy.yml` runs everything; `playbooks/unraid.yml`, `playbooks/proxmox.yml` and `playbooks/pve_host.yml` do one slice each (Unraid stacks, Proxmox guests, and the Proxmox node's backup config respectively). Within any of them, `--limit ` narrows by host and `-e only_stacks=[,...]` narrows by stack or app (name = the `name:` in the host's `stacks:`/`apps:` list, e.g. `forgejo` or `postgres` — not the `src:` path): ```sh # everything, both platforms ansible-playbook playbooks/deploy.yml # everything declared for nas2 ansible-playbook playbooks/unraid.yml --limit nas2 # just forgejo on nas2 ansible-playbook playbooks/unraid.yml --limit nas2 -e only_stacks=forgejo # forgejo and its database, skipping any other stack on nas2 ansible-playbook playbooks/unraid.yml --limit nas2 -e only_stacks=forgejo,postgres ``` Note `only_stacks` doesn't resolve dependencies for you — if you limit to `forgejo` before `postgres` has ever been deployed, the DB provisioning step will fail because there's nothing to connect to. CI doesn't hit this: it computes `only_stacks` from which `src/**` paths actually changed in the push, so an unrelated app's push never touches Postgres or Forgejo. `deploy.yml` runs Unraid before Proxmox by convention rather than necessity — the two platforms no longer depend on each other. Proxmox-side apps provision their databases against the Proxmox shared Postgres (192.168.50.54), and Terraform keeps its state on the CloudNativePG cluster on k3s; neither goes through Unraid. The node's backup config goes last, since a vzdump job pinned to specific VMIDs needs those guests to exist. Note that `only_stacks` has no meaning for `pve_host.yml` — it configures a node, not an app, and ignores it. `playbooks/k3s.yml` is separate from all of the above — it isn't included in `deploy.yml` and doesn't take `only_stacks`, because a cluster service there isn't a `stacks:` entry to filter down to. It has two selectors of its own instead: `--limit` narrows *which nodes*, and `--tags` narrows *which piece*. ```sh # the whole cluster: nodes, then the cluster services, then the kubeconfig ansible-playbook playbooks/k3s.yml # re-converge one worker, e.g. after reimaging it ansible-playbook playbooks/k3s.yml --limit k3s-node-02 # just one cluster service, no node install across 4 Pis ansible-playbook playbooks/k3s.yml --tags postgres # every cluster service, still no node install ansible-playbook playbooks/k3s.yml --tags services ``` The tags, one per play (see the playbook's header comment): `nodes` (both node plays), `metallb`, `monitoring`, `postgres`, `cert-manager`, `traefik`, `services` (all five of those together), `apps`, `upgrades`, `kubeconfig`. `services` and `apps` don't imply each other, so "everything except reinstalling k3s" is `--tags services,apps`. The SSH-key fetch play is tagged `always` rather than named, so it survives every filter — it's what any of the others connect with, not something you'd pick. Deploying a single service this way skips the `kubeconfig` play too, so `local/`'s copy isn't refreshed. That only matters on a cluster rebuild; add `--tags postgres,kubeconfig` if you want both. Worth knowing what a tagged run does and doesn't tell you: every service role just templates manifests into `/var/lib/rancher/k3s/server/manifests/` for k3s's own controllers to reconcile, so the playbook finishing means the files landed, not that the workload is up. Watch that separately, e.g. `kubectl -n shared-postgres get cluster,pods -w`. `playbooks/k3s_maintenance.yml` is separate again — see "Keeping it patched" above. It ignores `--limit` grouping in one sense worth knowing: `serial: 1` still applies to whatever `--limit` narrows the run to, so limiting to two nodes still patches them one at a time, not together. ## Adding a new app 1. `src//common/vars.yml` — `env_defaults:` for the config that doesn't depend on where it runs. 2. Build the platform side(s) you need: **Unraid** — `src//ansible/unraid/` containing `docker-compose.yml` (static, `${VAR}`-driven, joining `unraid_shared` with `external: true` if it needs the shared Postgres), `vars.yml` of overrides, and `.env.example` documenting every var with secrets blank. Optionally an `icon.png` and the `net.unraid.docker.*` labels, so the Docker page shows something other than a question mark. **Proxmox** — `src//ansible/proxmox/` containing `vars.yml`, `install.yml`, and templates for the app's config file and systemd unit; plus `src//terraform/` defining its LXC, called from `build/config/terraform/main.tf`. 3. Add its `homelab/` path to Vault (`docs/vault-secrets.md`). 4. Declare it on the target host — `stacks:` in an Unraid host's `host_vars/.yml`, or `apps:` in a Proxmox guest's. Same entry shape either way (name, src, vault_path, optional `db:` block). 5. Push — or run the relevant playbook manually before CI exists. 6. Leave the platform you didn't build as a README placeholder rather than deleting the folder. ## Removing an app (Unraid) **Deleting the entry from `stacks:` does not remove anything.** Ansible keeps no record of what it deployed last run, so an app that disappears from the list is simply never visited again — its containers keep running on the host, now unmanaged and invisible to the playbook. This is the one place the Unraid side doesn't behave like the Terraform side, where deleting a module call does destroy the LXC. Removal is therefore an instruction, not an absence. Set `state: absent` on the entry and leave it in place: ```yaml stacks: - name: arr src: arr vault_path: homelab/arr state: absent ``` The next run of `playbooks/unraid.yml` runs `docker compose down`, then deletes the project folder (and with it the rendered `.env`). Because a `host_vars/` edit is a change under `build/config/ansible/`, CI treats it as a full converge — so pushing that change is enough to action the teardown. Keep the entry as a tombstone until you're sure. It's the only record that the app was deliberately removed rather than never deployed, and flipping `state:` back to `present` redeploys it. ### What survives, and how to remove the rest Bind mounts are untouched, so `/mnt/user/appdata/` — which for a SQLite-backed app like Shelfarr *is* the application — survives a teardown and makes it reversible. External networks (`caddy-net`, `unraid_shared`) are left alone too; they belong to Unraid or to another stack. Both copies of the stack's icon *are* deleted without needing to be asked — the project-folder one goes with the folder, and the one under `/mnt/user/appdata/icons/` is removed explicitly. Neither is state; they're copies of a file in the repo, and a redeploy puts them back. Three things are destructive enough to stay opt-in, per entry: | Key | Effect | |---|---| | `remove_volumes: true` | Also delete the project's named volumes. Bind mounts are unaffected either way. | | `remove_images: local` | Also delete its images (`local` or `all`, as `docker compose down --rmi`). | | `remove_database: true` | Drop the Postgres database **and** its role. Only for stacks with a `db:` block. | Nothing removes the app's `homelab/` path from Vault — that has its own lifecycle, and destroying it would make the teardown irreversible. Delete it by hand with `vault kv metadata delete` once you're finished with the app. ### Order of operations `docker compose down` reads the compose file to know what it's removing, so the teardown has to run **before** you delete `src//` or the host's project folder. Delete those first and there's nothing left to tell Docker what belonged to the project — you're cleaning up by hand with `docker rm` and `docker network rm` instead. So: set `state: absent`, run the playbook, then delete files in a second pass. There's no equivalent on the Proxmox side yet. `lxc_app` only ever installs, and a symmetric teardown needs a per-app `uninstall.yml` convention that no app implements — the role is still a skeleton. For now, removing a Proxmox app means stopping and disabling its systemd unit by hand, or destroying the container via Terraform. ## Adding a host **Unraid:** add it under `unraid_servers` in `build/config/ansible/inventory/hosts.yml`, then create `build/config/ansible/inventory/host_vars/.yml` with its own `stacks:` list — each host only runs what it's assigned. **Proxmox:** define the LXC as a module in `src//terraform/`, call it from `build/config/terraform/main.tf`, and `terraform apply`. The guest appears in inventory automatically via the API — no `hosts.yml` edit — but it still needs a `host_vars/.yml` declaring its `apps:` list. **K3s:** add it as a new host under `k3s_control_plane` or `k3s_workers` (per `inventory/hosts.yml` → `k3s_cluster`) — no `terraform apply`, the Pi has to physically exist and have SSH access set up first (see "K3s (Raspberry Pi)" above). No `host_vars/` entry needed beyond that: `roles/k3s_node` reads `k3s_node_role` off the group, not the host, so which group a new Pi joins is the only thing that decides whether it becomes another agent or a second control-plane node — and this repo's role doesn't support the latter (single server, no HA etcd) without changes to `k3s_node/tasks/server.yml`.