# Homelab IaC — Project Notes Manages apps as infrastructure-as-code across three platforms — Unraid server(s), Proxmox, and a bare-metal k3s cluster — with secrets sourced from HashiCorp Vault and CI/CD via Forgejo Actions (self-hosted, on the Forgejo instance this repo deploys). Ansible is the deployment tool on **all three**; what differs is the deployment primitive and who creates the host: | | Unraid | Proxmox | k3s | |---|---|---|---| | Provision the host | already exists | Terraform (`bpg/proxmox`) creates an LXC | already exists (4 Raspberry Pis) | | Install the app | Ansible + `compose_stack` → `docker compose up -d` | Ansible + `lxc_app` → native install + systemd | Ansible + `k3s_app` → manifests into k3s's auto-deploy dir | | App config source | `common/vars.yml` + `ansible/unraid/vars.yml` | `common/vars.yml` + `ansible/proxmox/vars.yml` | `common/vars.yml` + `ansible/kubernetes/vars.yml` | | App list lives in | `host_vars/.yml` `stacks:` | `host_vars/.yml` `apps:` | `group_vars/k3s_cluster.yml` `k3s_apps:` | The k3s column is the newest and reached that shape late: `homelab-utils` started as cluster-bootstrap-only, with no `src//` entry, and the third column was added when Authentik moved onto it (see `docs/authentik-migration.md`). Two things about it don't reduce to the table: - **The cluster itself has no `src/` entry.** Terraform never provisions it and it isn't an app, so it gets an inventory group (`k3s_cluster`) and `roles/k3s_node`, the same way the Proxmox *node* gets `roles/pve_backup` while its guests get `lxc_app`. - **Cluster services aren't apps.** `k3s_metallb`, `k3s_monitoring`, `k3s_postgres` and `k3s_cert_manager` are the cluster's own infrastructure — one hard-coded role each, config in `group_vars`. Apps go through the generic `k3s_app` and live in `src/`. Both end up as `HelmChart` CRs, so they look alike on disk; the distinction is whether anything would still need it if no app were deployed. See "K3s cluster (homelab-utils)" below. ### Hosts Two Unraid entries are gone from the original plan: `nas2` was retired, and its address (192.168.50.2) was reassigned to `turtle-proxmox-01` — so a stale copy of it fails by reaching the wrong machine rather than by timing out. What remains: `nas01` (Unraid, 192.168.50.1), `turtle-proxmox-01`/`-02` (Proxmox, .2/.3), the guests on them from the API inventory, and the four k3s Pis (.60–.63, MetalLB pool .80–.99). Gateway is 192.168.50.254. **Proxmox LXC guests live in 192.168.50.50–.59** — Forgejo .52, shared Postgres .54. Addresses here are assigned by hand from that band, not derived from anything, so a new guest's `ip_address` is a question for whoever owns the network rather than a value to pick. The shared Postgres for everything outside the cluster is now the Proxmox LXC at 192.168.50.54 — see `docs/postgres-proxmox.md`. The Unraid Compose deployment under `src/shared/postgres/ansible/unraid/` is kept for the platform-shape reason every unused platform folder is kept, not because it is running anywhere. This repo supersedes the planning done in `../homelab/CLAUDE.md` — that file has the original decision log if you want the "why" behind the architecture in more detail. Multi-host support, the shared-Postgres pattern, and the Forgejo-Actions-hosts-itself bootstrap problem were decided after that doc was written; this file and `README.md` are the current source of truth. ## Layout - `src//common/vars.yml` — `env_defaults:` that hold regardless of where the app runs (version, ports, domain, DB name/user). - `src//ansible/unraid/` — the Compose deployment: `docker-compose.yml`, `.env.example`, a `vars.yml` of Unraid-specific overrides (appdata paths, the `shared-postgres` Docker network alias), and optionally an `icon.png` for the Unraid Docker page. - `src//ansible/proxmox/` — the native install: `vars.yml` of Proxmox-specific overrides, `install.yml` of app-specific install steps, and templates for the app's own config file and systemd unit. Two apps ship one: `src/forgejo/` (binary + systemd unit, the shape the role was written for) and `src/shared/postgres/` (distro packaging, so the unit comes from the package and `install.yml` only reconfigures the cluster). - `src//ansible/kubernetes/` — the k3s deployment: `vars.yml` of cluster-specific overrides, and one or more `*.yaml.j2` manifests (normally a `HelmChart` CR, plus anything the chart won't create — see `src/authentik/`). "kubernetes" rather than "k3s": the contents are plain Kubernetes apart from the `HelmChart` CR. - `src//terraform/` — a module defining this app's LXC (template, cores, memory, disk, IP), called from `build/config/terraform/main.tf`. - `src/shared//` — shared services (databases, etc.) used by multiple apps, same shape. - `build/config/ansible/` — inventory, playbooks, and the roles. Three of them deploy *apps*, one per platform, and share the same layering and `state: present|absent` contract: `compose_stack` (Unraid), `lxc_app` (Proxmox guests), `k3s_app` (the cluster). `compose_stack` and `k3s_app` each split into `deploy.yml`/`remove.yml`, dispatched by `main.yml` on the entry's `state:`. The rest configure infrastructure rather than apps: `k3s_node` (bootstraps a Pi as k3s server or agent, picked by inventory group), `k3s_metallb`, `k3s_monitoring`, `k3s_postgres`, `k3s_cert_manager` (cluster services), `k3s_maintenance` (drain/reboot/ uncordon), `unattended_upgrades`, and `pve_backup` (the Proxmox node's backup storage and vzdump schedule). - `build/config/terraform/` — Proxmox provider + `backend "pg"` state config, and the module calls saying what infrastructure exists. - `.forgejo/workflows/` — CI/CD, runs once Forgejo + a self-hosted runner exist (see "Bootstrapping" in README.md — chicken-and-egg on the first deploy). ## Key decisions - Compose files are static and reference `${VAR}`; only the rendered `.env` changes per deploy. Nothing secret is ever committed. - Config is layered, not duplicated: `common/vars.yml` merged with `ansible//vars.yml` (platform wins), then Vault secrets on top (secrets win). Both roles do this the same way, so moving an app between platforms means writing a new platform vars file, not restating its config. - One Vault KV path per app (`homelab/`), fetched as a whole dict via `community.hashi_vault.vault_kv2_get`. The prefix is `homelab/`, not `unraid/`, because an app's secrets don't change with the platform it lands on. - Unraid draws a Compose deployment's stack row and its containers from unrelated sources, so one committed `src//ansible/unraid/icon.png` is copied to two places. The **stack row** is Compose Manager serving `/icon.png` off disk — a file or nothing, no label and no URL, and only in the maintained fork (Compose Manager Plus); on the original plugin the copy is inert. The **containers** get icon, WebUI link and console shell from the dockerMan template that created them, which a Compose stack doesn't have, so each service carries `net.unraid.docker.*` labels (6.10+ falls back to them). The label path is a second copy under `/mnt/user/appdata/icons/` rather than the project-folder one, because the webgui resolves it on every Docker page load and that shouldn't hit the flash drive. - Of those labels only the icon is role-supplied: `compose_stack` injects `STACK_ICON` *underneath* both vars.yml layers, so an app can override it with a hosted URL. WebUI and shell stay literal in the compose file — the WebUI port has to be the published one and the shell is a property of the base image, so both belong next to `ports:` and `image:` rather than in a variable. Details, including where SVG does and doesn't work, in README.md → "Icons on the Unraid Docker page". - Shared Postgres is provisioned once per host; per-app DB/user creation is idempotent via `community.postgresql`, not `docker-entrypoint-initdb.d` (which only runs once on an empty volume — wrong fit for a shared service apps get added to over time). The Unraid instance was pinned to 13 because its data directory predated this repo; the Proxmox one is 17 from the PGDG archive, a fresh cluster with nothing to stay compatible with. - The Proxmox shared Postgres lives on the `AppData` ZFS pool with a `proxmox_replication` job to the second node, and **everything is on one volume** — no `mount_point` splitting PGDATA from the WAL. Unlike Forgejo's "no bind mounts", which is about what `vzdump` includes, this is a correctness requirement: replication snapshots each of a guest's volumes independently, so two volumes replicate as two snapshots taken at different instants, and the far node can end up with a data directory and a WAL that disagree. One volume means one atomic snapshot — exactly the crash-consistent image WAL replay is designed for. It also means running out of space has one fix and not two: raise `disk_size` on the module call (256 GiB as of this writing, up from the original 32 once Forgejo's database made the OS-sized rootfs look optimistic), never add a second disk. Cheap to do — on ZFS the resize is a refquota change applied to a running guest — but one-way, since shrinking is a replacement and `prevent_destroy` blocks it. Runbook in `docs/postgres-proxmox.md`. Two further consequences worth keeping straight: replication is configured *per guest*, not per pool, so a container created on `AppData` without a job is a single point of failure that looks identical in the storage view (hence Terraform owning the job rather than a hand-run `pvesr`); and it's asynchronous, so a failover loses up to `replication_schedule` — five minutes — of commits. It is not a backup; `pve_backup`'s vzdump is. - `full_page_writes = off` on the Proxmox instance, and only there. It guards against a torn page, which copy-on-write storage cannot produce, so on ZFS the full-page images are pure write amplification that also becomes replication traffic. The same setting on ext4 or xfs risks unrecoverable corruption — which is why it's in `ansible/proxmox/vars.yml` and must never migrate to `common/vars.yml`. - Terraform provisions the guest and stops there — it has no way to deploy an app into an LXC, so Ansible picks up from the point the container exists. - Proxmox guests keep all their state on their own rootfs; no bind mounts, no NFS shares mounted into a container. Counter-intuitive if the goal is "back the repos up to the NAS", but `vzdump` excludes bind mounts by design, so a bind-mounted repo directory is the one thing *missing* from the archive. Everything local means one archive is a complete filesystem restore point. Getting a copy off-box is the node's job (`pve_backup` → NFS storage on the NAS + a vzdump schedule), not the container's. - State an app keeps *outside* its container is that app's problem. Forgejo's database is on the shared Postgres LXC — a different guest, and so a different vzdump archive — so a systemd timer in the container `pg_dump`s it onto the container's own disk, and the vzdump archive carries both halves. The dump schedule and the vzdump window are a pair — move one, move the other. - Proxmox hosts come from the `community.proxmox` dynamic inventory plugin querying the Proxmox API, so `terraform apply` creating an LXC doesn't also require an inventory edit. - The `terraform` tag is what marks a Proxmox guest as this repo's to manage. The dynamic inventory returns every LXC on the cluster, hand-made ones included, so `inventory/proxmox.yml` filters on that tag — both Terraform modules set it (`tags` in their `variables.tf`), nothing else does. It filters rather than narrowing the `proxmox_guests` condition, so an unmanaged guest is absent from `all` as well and no future play can reach it by accident. The failure mode is silent in one direction: a new module that forgets the tag creates a container Ansible simply never visits, with no error anywhere — same shape as an omitted `stacks:` entry, and the first thing to check if a freshly-applied guest is never deployed to. - Terraform state lives in Postgres (`backend "pg"`), not a local file, so CI and a workstation see the same state — specifically the **CloudNativePG cluster on k3s**, not the shared Postgres this configuration provisions on Proxmox. That would be circular: Terraform would need the database to exist in order to create the container the database runs in. State goes somewhere Terraform has no hand in building, which breaks the cycle outright at the cost of a dependency the rest of the repo avoids — `terraform apply` now needs the Pis up. Worth it over the alternative (local state, apply, `init -migrate-state`), which has to be got right exactly once. Bootstrap ordering is `docs/postgres-proxmox.md`. - Every app keeps the same `common/` + `ansible/` + `terraform/` shape even when only one platform is in use, so adding the other later doesn't require restructuring. - Removal on Unraid is explicit (`state: absent` on the stack entry), never implied by deleting the entry. Ansible stores no record of the last run, so an omitted stack isn't removed — it's just never visited again, and keeps running unmanaged. The entry stays as a tombstone documenting that the app was deliberately removed. Volumes, images and the app's database are further opt-ins (`remove_volumes`, `remove_images`, `remove_database`) rather than being implied by `state: absent`, and the Vault path is never touched — a default teardown is meant to be reversible. `lxc_app` has no counterpart yet; it would need a per-app `uninstall.yml`. - The k3s cluster's join token is a fixed value from Vault (`homelab/k3s-homelab-utils` → `K3S_TOKEN`), not the one `k3s server` generates for itself on first install. Both the control-plane play and the worker play look it up independently, so a full rebuild — wipe both SD cards, reinstall — reproduces the same cluster identity instead of needing a freshly-generated token fetched off the server and redistributed by hand. Same "generate once, store it, never let a redeploy invent a new one" reasoning as Forgejo's fixed secrets, applied to cluster identity instead of app state. - The SSH key the k3s plays connect with is fetched from Vault (`homelab/ci/ssh-k3s`) by `playbooks/k3s_ssh_key.yml` and written to `local/`, which both `k3s.yml` and `k3s_maintenance.yml` import as their first play — rather than the operator loading it into `ssh-agent` first, as the Unraid manual flow still expects. Same arrangement CI already uses for the Unraid key, just moved into the playbook because these two are manual-only and there's no workflow to hold the fetch step. It's a separate playbook, not a `pre_tasks` block, because both entry points need it and neither can assume the other ran. It targets `k3s_cluster` rather than `localhost` for a non-obvious reason: the implicit localhost isn't a member of `all`, so it wouldn't inherit `vault_addr`/`vault_kv_mount`/ `vault_auth_method` from `group_vars/all.yml`. Nothing in it connects to a Pi — `gather_facts: false` plus `delegate_to: localhost`, necessarily, since the key isn't on disk yet at that point. Falling back to a local key takes *two* overrides (`-e k3s_ssh_key_fetch=false` alongside `-e ansible_ssh_private_key_file=...`): pointing at another key alone doesn't help if Vault is what's broken, because the fetch play fails before any other play runs. - `playbooks/k3s.yml` is deliberately **not** wired into `deploy.yml` or `.forgejo/workflows/deploy.yml`. Converging 4 physical nodes on every push is a bigger blast radius than restarting a Compose stack — the same reasoning that keeps `terraform apply` manual for Proxmox. Run it by hand. - `playbooks/k3s.yml`'s plays are tagged so one piece can be converged on its own — `nodes`, `metallb`, `monitoring`, `postgres`, `services` (those three cluster services together), `upgrades`, `kubeconfig`. Play-level tags, one per play, not per-task tags inside the roles: the plays already are the units, and tagging tasks would mean every role carrying tags for a selectivity only this playbook needs. This is the k3s equivalent of `only_stacks` for Unraid — a cluster service isn't a `stacks:` entry, so `only_stacks` can't reach it. The SSH-key import play is tagged `always` rather than getting a name of its own, since it's the prerequisite for anything else connecting and has to survive every `--tags` filter. Note what a tagged service run does *not* cover: the roles only drop manifests for k3s's controllers to reconcile, so a green playbook means the files landed, not that the workload came up. - `k3s_node` is one role, not two, even though a server and an agent install differently. Both need the same OS prep (cgroups, swap), and which install path runs is a group var (`k3s_node_role: server`/`agent`, set per inventory group) rather than a second role — so the shared prep can't drift between the two nodes types the way copy-pasted tasks eventually would. - Extra `INSTALL_K3S_EXEC` flags split across two vars, not one: `k3s_extra_args` (both roles) and `k3s_server_extra_args` (control-plane only). `k3s agent` doesn't understand server-only flags like `--disable` and fails to start if handed one, so a flag like `--disable=servicelb` (which `k3s_metallb` needs to avoid fighting k3s's bundled ServiceLB for the same IPs) can't safely live in the shared list. - `k3s_node`'s install tasks track the exec line they last installed a node with (a small marker file next to k3s's own config) and reinstall — restarting the `k3s`/`k3s-agent` service, not the node — whenever it drifts from what `k3s_extra_args`/`k3s_server_extra_args`/`k3s_api_tls_san` currently say, even if `k3s_version` hasn't changed. Deliberately folded into the ordinary install task rather than a separate maintenance playbook: `playbooks/k3s.yml` is already manual-only and documented as safe to re-run, and unlike the reboots `k3s_maintenance` guards, bouncing the k3s process doesn't take pods down — containerd keeps them running underneath it. The worker play still runs with `serial: 1` so at most one node's kubelet is ever mid-restart at once. - `k3s server`/`k3s agent` are installed with an explicit `--node-name {{ inventory_hostname }}`, not left to default to the OS hostname. This is what lets `roles/k3s_maintenance` address a node by `inventory_hostname` when draining/uncordoning — the k8s node object and the Ansible host are guaranteed to be the same string. - Patching a k3s Pi is split into two roles that don't know about each other directly, only through a file: `unattended_upgrades` installs updates hands-off but with `Automatic-Reboot "false"`, and `k3s_maintenance` is the only thing that actually reboots a node — triggered by the presence of `/var/run/reboot-required`, which is the OS's own signal, not something either role invents. Splitting it this way means the risky half (taking a node out of the cluster) is exactly one thing (`playbooks/k3s_maintenance.yml`, `serial: 1`), not entangled with the routine half (installing packages), which runs unattended every day on every node without anyone thinking about it. - `k3s_maintenance`'s kubectl calls (`drain`, `wait`, `uncordon`) are delegated to the control-plane host and run as `k3s kubectl`, k3s's own bundled client, rather than requiring a kubectl install or a kubeconfig anywhere else — including on whatever eventually runs this on a schedule. This also has to work when the node currently being processed *is* the control plane: delegating a host to itself is just a normal SSH connection, used right up until the moment that connection reboots out from under it. - `k3s_monitoring` (a lean kube-prometheus-stack, for OpenLens and similar tools) is installed as a `HelmChart` custom resource dropped into k3s's own auto-deploying manifests directory, not run through a `helm` binary or the `kubernetes.core.helm` collection — k3s ships a helm-controller that reconciles anything found there, the same mechanism it uses to install its own bundled Traefik and ServiceLB. That keeps the pattern this repo already uses everywhere else (Ansible renders a file, something else converges on it) instead of adding a second, unrelated way to reach the cluster from the controller. Grafana and Alertmanager are left disabled — OpenLens brings its own dashboards and this cluster doesn't page anyone — and the control-plane scrape targets (`kubeControllerManager`, `kubeScheduler`, `kubeProxy`, `kubeEtcd`) are disabled too, since k3s bundles those into one static binary instead of exposing them the way the chart expects; leaving them on just produces permanently-"down" targets, not a working scrape. - `k3s_metallb` (MetalLB, L2 mode) follows the same `HelmChart` CR pattern as `k3s_monitoring`, plus a second, plain manifest for its `IPAddressPool`/`L2Advertisement` config dropped in the same auto-deploying directory — k3s's deploy controller applies any manifest it finds there, not only `HelmChart` CRs, and retries one referencing CRDs that don't exist yet rather than failing outright, so the config doesn't need to wait on the chart install finishing first. It replaces k3s's bundled ServiceLB rather than running next to it — both would otherwise hand out IPs for the same `LoadBalancer` Services — so `k3s_server_extra_args` in `inventory/group_vars/k3s_cluster.yml` carries `--disable=servicelb` (server-only — `k3s_extra_args`, applied to both server and agent, can't carry a server-only flag like `--disable` without breaking agent installs). The IP pool (`k3s_metallb_address_range`, same file) has no built-in default; the role fails fast rather than silently advertising an empty pool. - `k3s_postgres` (CloudNativePG) is the k3s cluster's own shared Postgres — a third platform for `src/shared/postgres/`, alongside Unraid and Proxmox, deployed via Helm rather than Compose or a native install. Chosen over Bitnami's postgresql-ha (repmgr+pgpool, and Bitnami's free chart/image catalog was restructured into a "legacy" repo in 2025) and the Zalando operator (Patroni-based, heavier) because it fits the same `HelmChart` CR pattern already established: the operator installs as a chart CR, same shape as `k3s_metallb`/`k3s_monitoring`, and the actual cluster is a plain `Cluster` CR manifest — same "config manifest that outlives the CRDs it references" trick as `k3s_metallb`'s `IPAddressPool`. It's a separate physical instance from the Docker-based shared/postgres (pinned to 13), but reuses the same Vault path (`homelab/shared/postgres`) rather than a k3s-specific one — one superuser identity for the "shared postgres" concept regardless of which platform it's running on, consistent with how every host already draws from that same path. 1 primary + 1 replica, scheduled on worker Pis only via `nodeAffinity` — the control plane stays free of app pods, same boundary `k3s_metallb`/`k3s_monitoring` already keep. HA replication only for now, no backups. - The CNPG cluster is published on the LAN through MetalLB at a pinned `k3s_postgres_loadbalancer_ip`, on top of the three `ClusterIP` Services CNPG makes for every `Cluster`. Declared inside the `Cluster` CR as a managed service (`.spec.managed.services.additional`, `selectorType: rw`) rather than as a Service manifest of this repo's own, so the operator owns the selector and the address follows a failover instead of needing to be re-pointed by hand. Pinned rather than auto-assigned because the address is meant to be written down in config elsewhere; it comes out of the bottom of `k3s_metallb_address_range` and would need a second `autoAssign: false` pool if anything else on this cluster ever wants a fixed IP. This is what closes the reachability half of per-app DB provisioning — the `community.postgresql` tasks `compose_stack`/`lxc_app` use can now reach this instance the same way they reach the Unraid one. `roles/k3s_app` is the other half, and uses exactly those tasks; note the asymmetry it relies on, which is easy to misread as a mistake. The *provisioning* runs from the Ansible controller and therefore uses the LoadBalancer address (`k3s_postgres_loadbalancer_ip`), because `delegate_to: localhost` is off the cluster and can't route to a ClusterIP. The *app* uses the in-cluster `shared-postgres-rw` DNS name, because sending pod traffic out to the LAN and back would put MetalLB's L2 speaker in the path of every query for no benefit. Two addresses for one database, each correct for its caller. - `roles/k3s_app` is the cluster's counterpart to `compose_stack` and `lxc_app`: the generic role that deploys an *app*, driven by `k3s_apps:` in `group_vars/k3s_cluster.yml`, as distinct from the `k3s_*` roles that each deploy one cluster service. It renders whatever `*.yaml.j2` the app ships under `src//ansible/kubernetes/` into k3s's auto-deploying manifests directory, so adding a manifest to an app is dropping a file next to the others rather than a role change. Secrets go into a separate Kubernetes Secret rendered from Vault, never interpolated into the manifests — the same committed-config/generated-secrets split as Unraid's static compose file plus rendered `.env`, which is why the manifests can stay 0644 on the node while the Secret is 0600. - `k3s_apps:` lives in `group_vars/k3s_cluster.yml`, not `host_vars/`, unlike both other platforms. An app is deployed to the cluster, not to a node: the play runs against `k3s_control_plane` only because that's where the manifests directory is, which is an implementation detail of how k3s is reached rather than a statement about where the app runs. - Removal on k3s is one pass, not two. `compose_stack` needs the compose file still on disk to run `docker compose down`, so `state: absent` and deleting `src//` have to be separate commits (see "Removing an app" below). k3s's deploy controller tracks which resources each manifest file created, so deleting the file *is* the teardown and `src//` can go in the same commit. The tombstone convention still applies for the same reason as everywhere else — an entry dropped from `k3s_apps:` is never visited again and keeps running unmanaged. - `k3s_cert_manager` is the fourth cluster service and the first that exists purely for apps rather than for the cluster — nothing in the cluster needs a certificate, `k3s_app`'s tenants do. It's still a cluster service rather than an app, on the same reasoning as the shared Postgres: one `ClusterIssuer` that every app's Ingress annotates itself against, rather than each app carrying its own ACME account and DNS credentials. DNS-01, not HTTP-01, because HTTP-01 needs Let's Encrypt to reach the cluster from the internet on port 80 and this LAN deliberately isn't reachable — which has the useful side effect that a certificate can be issued *before* DNS points at the cluster, so a migration's TLS is settled before its cutover. The solver stanza itself has no default and the role fails fast without one: it depends on who runs the DNS, which the repo can't know. Same fail-fast-on-unset treatment as `k3s_metallb_address_range`. - Apps on k3s reach the outside through the Traefik k3s already bundles, on the MetalLB address its Service holds, rather than each getting its own LoadBalancer IP. One entrypoint and host-based routing means adding an app costs a DNS record, not a pool address — the opposite of the choice made for `k3s_postgres`, which is pinned to its own IP precisely because Postgres isn't HTTP and can't be name-routed. - `k3s_traefik` is the one cluster-service role that installs nothing: k3s installs Traefik itself, so the role only adjusts it, via a `HelmChartConfig` merged over k3s's own `HelmChart` rather than an edit to the `traefik.yaml` k3s rewrites on every server start. Its scope is the dashboard, which a stock k3s 404s — Traefik still builds it, but the chart stopped shipping the router that reaches it in v28. The role restores that router on Traefik's internal `traefik` entrypoint (port 9000, unpublished → port-forward only), and optionally publishes the dashboard on a hostname behind an Authentik forward-auth middleware when `k3s_traefik_dashboard_host` is set. Three things worth keeping straight: the internal route stays on even when the hostname one exists, because authenticated access depends on Authentik → CNPG → a healthy cluster, exactly what you'd open the dashboard to diagnose; the published route is an `IngressRoute` with an explicit cert-manager `Certificate` rather than an annotated `Ingress`, because the dashboard is `api@internal` and has no Service to point an `Ingress` at; and the role fails fast if a hostname is set without an auth address, since the failure mode is publishing every router, service and middleware on the cluster to the LAN unauthenticated. Authentik's provider must be **forward auth (domain level)** — single-application mode needs `/outpost.goauthentik.io/` routed on the dashboard's own host, a cross-namespace service reference Traefik rejects unless `allowCrossNamespace` is on. ## Adding a new app 1. `src//common/vars.yml` — `env_defaults:` for the portable config. 2. Whichever platform(s) it targets: - **Unraid:** `src//ansible/unraid/` with `docker-compose.yml` (static, `${VAR}`-driven), `vars.yml` of overrides, `.env.example` documenting every var. - **Proxmox:** `src//ansible/proxmox/` with `vars.yml`, `install.yml`, and config/systemd templates; plus `src//terraform/` defining its LXC, wired into `build/config/terraform/main.tf`. - **k3s:** `src//ansible/kubernetes/` with `vars.yml` and at least one `*.yaml.j2` manifest — normally a `HelmChart` CR, plus anything the chart won't create for you. See `src/authentik/` for a worked example, including the two things a chart usually leaves out: a PVC, and secrets referenced by `secretKeyRef` rather than set as values. 3. Populate its Vault path (`homelab/`) with real secret values. 4. Declare it on the target: - Unraid → `host_vars/.yml` `stacks:` list. - Proxmox → `host_vars/.yml` `apps:` list. - k3s → `group_vars/k3s_cluster.yml` `k3s_apps:` list. All three take the same entry shape (name, src, vault_path, optional `db:`, optional `state:`). Never declare the same app on more than one at once — they share a Vault path and a database, so two live deployments corrupt each other's state. Moving an app *between* platforms is therefore a cutover with a verification step in the middle, not an edit; `docs/authentik-migration.md` is the worked example. 5. Leave the unused platforms' folders as README placeholders rather than deleting them — the shape is the point. ## Removing an app **Unraid.** Set `state: absent` on its `stacks:` entry and run `playbooks/unraid.yml` — don't delete the entry, which removes nothing. Then delete `src//` in a *second* pass, never before: `docker compose down` needs the compose file to know what it's tearing down. Full detail, including the opt-in flags for volumes/images/database, is in README.md → "Removing an app". **k3s.** Same `state: absent` tombstone rule, but one pass — deleting the manifests is the teardown, because k3s's deploy controller garbage-collects what each file created, so `src//` can go in the same commit. The database, PVCs and Vault path deliberately survive; there's no `remove_volumes`/`remove_database` equivalent yet, and adding one is a "when it's actually wanted" job rather than a guess at the shape. **Proxmox.** No teardown path at all — `lxc_app` would need a per-app `uninstall.yml`, and neither app that ships an `install.yml` has one. Note this is worse for the shared Postgres than for an ordinary app: tearing it down means every other app's database as well. ## K3s cluster (homelab-utils) 4 Raspberry Pis, `inventory/hosts.yml` → `k3s_cluster` (`k3s_control_plane`: 1 host, `k3s_workers`: 3). No Terraform — the Pis already exist. The cluster itself has no `src//` entry (it isn't an app), but apps now land on it: `k3s_apps:` in `group_vars/k3s_cluster.yml`, deployed by `roles/k3s_app` from `src//ansible/kubernetes/`. `playbooks/k3s.yml` is manual-only (see "Key decisions" above); run it with: ```sh cd build/config/ansible ansible-playbook playbooks/k3s.yml # or one piece of it — see "Key decisions" above for the tag list ansible-playbook playbooks/k3s.yml --tags postgres # apps only, or one app ansible-playbook playbooks/k3s.yml --tags apps ansible-playbook playbooks/k3s.yml --tags apps -e only_apps=authentik ``` `--tags services` covers the cluster's own infrastructure and `--tags apps` what runs on top of it; neither implies the other, so "everything except reinstalling k3s" is `--tags services,apps`. Prerequisites, once, before the first run: the `ansible` user exists on every Pi with NOPASSWD sudo and the **public** half of this repo's k3s SSH key installed, and `homelab/k3s-homelab-utils` → `K3S_TOKEN` plus `homelab/ci/ssh-k3s` → `PRIVATE_KEY` are set in Vault. Nothing needs setting up on the controller beyond `VAULT_ADDR`/`VAULT_TOKEN` — the private key comes from Vault at run time (see "Key decisions" above). Full detail in README.md → "K3s (Raspberry Pi)" and `docs/vault-secrets.md`. **Rebuilding**: reimage the SD card(s), redo the `ansible` user + SSH key step, then re-run the playbook — the fixed Vault token means the rebuilt node(s) rejoin the same cluster identity rather than needing a token hunted down from a live server. **Adding a fifth Pi**: add it under `k3s_control_plane` or `k3s_workers` in `hosts.yml`; no `host_vars/` entry needed, `k3s_node_role` comes from the group. This repo's `k3s_node/server.yml` assumes a single control-plane node — adding a second one for HA needs changes there, not just an inventory edit. **Patching**: `playbooks/k3s.yml` also applies `roles/unattended_upgrades` to every node, so updates install themselves daily with no login required — but never reboot themselves (`Automatic-Reboot "false"`). Run `playbooks/k3s_maintenance.yml` to actually apply a pending reboot: it checks `/var/run/reboot-required` per node and, only where set, drains → reboots → waits for `Ready` → uncordons, `serial: 1` so only one node is ever down. Manual-only for now, same as `playbooks/k3s.yml` — meant to eventually run on a schedule from a self-hosted Forgejo Actions runner rather than by hand. **Metrics**: `playbooks/k3s.yml` also applies `roles/k3s_monitoring`, a lean kube-prometheus-stack (no Grafana/Alertmanager) so OpenLens and similar tools can show node/pod metrics — see "Key decisions" above for why it's a `HelmChart` CR rather than a `helm` install, and README.md → "Metrics (Prometheus)" for pointing OpenLens at it. **LoadBalancer IPs**: `playbooks/k3s.yml` also applies `roles/k3s_metallb`, MetalLB in L2 mode, so `type: LoadBalancer` Services get real LAN IPs instead of staying `ClusterIP`-only. Set the pool it hands out from — `k3s_metallb_address_range` in `inventory/group_vars/k3s_cluster.yml` — to a range your DHCP scope and every static assignment in `hosts.yml` steer clear of before the first run; see "Key decisions" above for why it replaces ServiceLB rather than running alongside it, and README.md → "LoadBalancer IPs (MetalLB)" for more. There's no *node* teardown path, unlike `state: absent` for apps — removing a Pi today means wiping its SD card and deleting its inventory entry by hand. **Shared Postgres**: `playbooks/k3s.yml` also applies `roles/k3s_postgres`, CloudNativePG — the cluster's own shared Postgres, a third platform for `src/shared/postgres/` alongside Unraid and Proxmox. See "Key decisions" above for why CloudNativePG and the topology, and README.md → "Postgres (CloudNativePG)" for connecting to it. Per-app database provisioning works now (`roles/k3s_app`, `db:` on a `k3s_apps:` entry) — note the two-address asymmetry described in "Key decisions". Still **no backups**: HA replication only, which protects against a dead SD card and not against a dropped table. That gap matters more now that Authentik's database lives here rather than inside whatever covers Unraid's `/mnt/user/appdata`; a CNPG `ScheduledBackup` is the obvious next piece of work. **TLS**: `playbooks/k3s.yml` also applies `roles/k3s_cert_manager` — cert-manager plus one `ClusterIssuer` that every app's Ingress annotates itself against. Two values have no default and the role refuses to run without them: `k3s_cert_manager_acme_email` and `k3s_cert_manager_solver` (the DNS-01 stanza, which depends on your DNS provider — a commented Cloudflare example is in `group_vars/k3s_cluster.yml`). Use the Let's Encrypt staging directory while working a solver out; production allows five failed validations per hostname per hour. **Traefik dashboard**: `playbooks/k3s.yml` also applies `roles/k3s_traefik` — the only cluster-service role that installs nothing, since k3s installs Traefik itself. Out of the box the dashboard is reachable only by port-forward (`kubectl -n kube-system port-forward deploy/traefik 9000:9000`, then `http://127.0.0.1:9000/dashboard/` — trailing slash required; without this role a stock k3s 404s there). Publishing it on a hostname behind Authentik is one commented-out line, `k3s_traefik_dashboard_host` in `group_vars/k3s_cluster.yml`, plus a domain-level forward-auth Proxy Provider on the Authentik side. See "Key decisions" above and README.md → "Dashboard (Traefik)". **Apps**: `k3s_apps:` in `group_vars/k3s_cluster.yml`, deployed by `roles/k3s_app`. Authentik is the first and currently only one — it moved off a hand-made Unraid container rather than being deployed fresh, so `src/authentik/ansible/unraid/` is a README explaining why there's no compose file there. The cutover procedure is `docs/authentik-migration.md`; the part worth knowing without reading it is that the deployed `AUTHENTIK_VERSION` must match the version a restored database was dumped from, because Authentik's migrations run on startup and are one-way.