homelab/CLAUDE.md

# Homelab IaC — Project Notes

Manages apps as infrastructure-as-code across three platforms — Unraid
server(s), Proxmox, and a bare-metal k3s cluster — with secrets sourced from
HashiCorp Vault and CI/CD via Forgejo Actions (self-hosted, on the Forgejo
instance this repo deploys).

Ansible is the deployment tool on **all three**; what differs is the
deployment primitive and who creates the host:

| | Unraid | Proxmox | k3s |
|---|---|---|---|
| Provision the host | already exists | Terraform (`bpg/proxmox`) creates an LXC | already exists (4 Raspberry Pis) |
| Install the app | Ansible + `compose_stack` → `docker compose up -d` | Ansible + `lxc_app` → native install + systemd | Ansible + `k3s_app` → manifests into k3s's auto-deploy dir |
| App config source | `common/vars.yml` + `ansible/unraid/vars.yml` | `common/vars.yml` + `ansible/proxmox/vars.yml` | `common/vars.yml` + `ansible/kubernetes/vars.yml` |
| App list lives in | `host_vars/<host>.yml` `stacks:` | `host_vars/<guest>.yml` `apps:` | `group_vars/k3s_cluster.yml` `k3s_apps:` |

The k3s column is the newest and reached that shape late: `homelab-utils`
started as cluster-bootstrap-only, with no `src/<app>/` entry, and the third
column was added when Authentik moved onto it (see
`docs/authentik-migration.md`). Two things about it don't reduce to the
table:

- **The cluster itself has no `src/` entry.** Terraform never provisions it
  and it isn't an app, so it gets an inventory group (`k3s_cluster`) and
  `roles/k3s_node`, the same way the Proxmox *node* gets `roles/pve_backup`
  while its guests get `lxc_app`.
- **Cluster services aren't apps.** `k3s_metallb`, `k3s_monitoring`,
  `k3s_postgres` and `k3s_cert_manager` are the cluster's own
  infrastructure — one hard-coded role each, config in `group_vars`. Apps go
  through the generic `k3s_app` and live in `src/`. Both end up as
  `HelmChart` CRs, so they look alike on disk; the distinction is whether
  anything would still need it if no app were deployed. See "K3s cluster
  (homelab-utils)" below.

### Hosts

Two Unraid entries are gone from the original plan: `nas2` was retired, and its
address (192.168.50.2) was reassigned to `turtle-proxmox-01` — so a stale copy
of it fails by reaching the wrong machine rather than by timing out. What
remains: `nas01` (Unraid, 192.168.50.1), `turtle-proxmox-01`/`-02` (Proxmox,
.2/.3), the guests on them from the API inventory, and the four k3s Pis
(.60–.63, MetalLB pool .80–.99). Gateway is 192.168.50.254.

**Proxmox LXC guests live in 192.168.50.50–.59** — Forgejo .52, shared
Postgres .54. Addresses here are assigned by hand from that band, not derived
from anything, so a new guest's `ip_address` is a question for whoever owns
the network rather than a value to pick.

The shared Postgres for everything outside the cluster is now the Proxmox LXC
at 192.168.50.54 — see `docs/postgres-proxmox.md`. The Unraid Compose
deployment under `src/shared/postgres/ansible/unraid/` is kept for the
platform-shape reason every unused platform folder is kept, not because it is
running anywhere.

This repo supersedes the planning done in `../homelab/CLAUDE.md` — that file
has the original decision log if you want the "why" behind the architecture
in more detail. Multi-host support, the shared-Postgres pattern, and the
Forgejo-Actions-hosts-itself bootstrap problem were decided after that doc
was written; this file and `README.md` are the current source of truth.

## Layout

- `src/<app>/common/vars.yml` — `env_defaults:` that hold regardless of where
  the app runs (version, ports, domain, DB name/user).
- `src/<app>/ansible/unraid/` — the Compose deployment: `docker-compose.yml`,
  `.env.example`, a `vars.yml` of Unraid-specific overrides (appdata paths,
  the `shared-postgres` Docker network alias), and optionally an `icon.png`
  for the Unraid Docker page.
- `src/<app>/ansible/proxmox/` — the native install: `vars.yml` of
  Proxmox-specific overrides, `install.yml` of app-specific install steps,
  and templates for the app's own config file and systemd unit. Two apps
  ship one: `src/forgejo/` (binary + systemd unit, the shape the role was
  written for) and `src/shared/postgres/` (distro packaging, so the unit
  comes from the package and `install.yml` only reconfigures the cluster).
- `src/<app>/ansible/kubernetes/` — the k3s deployment: `vars.yml` of
  cluster-specific overrides, and one or more `*.yaml.j2` manifests (normally
  a `HelmChart` CR, plus anything the chart won't create — see
  `src/authentik/`). "kubernetes" rather than "k3s": the contents are plain
  Kubernetes apart from the `HelmChart` CR.
- `src/<app>/terraform/` — a module defining this app's LXC (template, cores,
  memory, disk, IP), called from `build/config/terraform/main.tf`.
- `src/shared/<service>/` — shared services (databases, etc.) used by
  multiple apps, same shape.
- `build/config/ansible/` — inventory, playbooks, and the roles. Three of
  them deploy *apps*, one per platform, and share the same layering and
  `state: present|absent` contract: `compose_stack` (Unraid), `lxc_app`
  (Proxmox guests), `k3s_app` (the cluster). `compose_stack` and `k3s_app`
  each split into `deploy.yml`/`remove.yml`, dispatched by `main.yml` on the
  entry's `state:`. The rest configure infrastructure rather than apps:
  `k3s_node` (bootstraps a Pi as k3s server or agent, picked by inventory
  group), `k3s_metallb`, `k3s_monitoring`, `k3s_postgres`,
  `k3s_cert_manager` (cluster services), `k3s_maintenance` (drain/reboot/
  uncordon), `unattended_upgrades`, and `pve_backup` (the Proxmox node's
  backup storage and vzdump schedule).
- `build/config/terraform/` — Proxmox provider + `backend "pg"` state config,
  and the module calls saying what infrastructure exists.
- `.forgejo/workflows/` — CI/CD, runs once Forgejo + a self-hosted runner
  exist (see "Bootstrapping" in README.md — chicken-and-egg on the first
  deploy).

## Key decisions

- Compose files are static and reference `${VAR}`; only the rendered `.env`
  changes per deploy. Nothing secret is ever committed.
- Config is layered, not duplicated: `common/vars.yml` merged with
  `ansible/<platform>/vars.yml` (platform wins), then Vault secrets on top
  (secrets win). Both roles do this the same way, so moving an app between
  platforms means writing a new platform vars file, not restating its config.
- One Vault KV path per app (`homelab/<app>`), fetched as a whole dict via
  `community.hashi_vault.vault_kv2_get`. The prefix is `homelab/`, not
  `unraid/`, because an app's secrets don't change with the platform it lands
  on.
- Unraid draws a Compose deployment's stack row and its containers from
  unrelated sources, so one committed `src/<app>/ansible/unraid/icon.png` is
  copied to two places. The **stack row** is Compose Manager serving
  `<project dir>/icon.png` off disk — a file or nothing, no label and no URL,
  and only in the maintained fork (Compose Manager Plus); on the original
  plugin the copy is inert. The **containers** get icon, WebUI link and
  console shell from the dockerMan template that created them, which a Compose
  stack doesn't have, so each service carries `net.unraid.docker.*` labels
  (6.10+ falls back to them). The label path is a second copy under
  `/mnt/user/appdata/icons/` rather than the project-folder one, because the
  webgui resolves it on every Docker page load and that shouldn't hit the
  flash drive.
- Of those labels only the icon is role-supplied: `compose_stack` injects
  `STACK_ICON` *underneath* both vars.yml layers, so an app can override it
  with a hosted URL. WebUI and shell stay literal in the compose file — the
  WebUI port has to be the published one and the shell is a property of the
  base image, so both belong next to `ports:` and `image:` rather than in a
  variable. Details, including where SVG does and doesn't work, in README.md →
  "Icons on the Unraid Docker page".
- Shared Postgres is provisioned once per host; per-app DB/user creation
  is idempotent via `community.postgresql`, not `docker-entrypoint-initdb.d`
  (which only runs once on an empty volume — wrong fit for a shared service
  apps get added to over time). The Unraid instance was pinned to 13 because
  its data directory predated this repo; the Proxmox one is 17 from the PGDG
  archive, a fresh cluster with nothing to stay compatible with.
- The Proxmox shared Postgres lives on the `AppData` ZFS pool with a
  `proxmox_replication` job to the second node, and **everything is on one
  volume** — no `mount_point` splitting PGDATA from the WAL. Unlike Forgejo's
  "no bind mounts", which is about what `vzdump` includes, this is a
  correctness requirement: replication snapshots each of a guest's volumes
  independently, so two volumes replicate as two snapshots taken at different
  instants, and the far node can end up with a data directory and a WAL that
  disagree. One volume means one atomic snapshot — exactly the
  crash-consistent image WAL replay is designed for. It also means running
  out of space has one fix and not two: raise `disk_size` on the module call
  (256 GiB as of this writing, up from the original 32 once Forgejo's
  database made the OS-sized rootfs look optimistic), never add a second
  disk. Cheap to do — on ZFS the resize is a refquota change applied to a
  running guest — but one-way, since shrinking is a replacement and
  `prevent_destroy` blocks it. Runbook in `docs/postgres-proxmox.md`. Two
  further consequences worth keeping straight: replication is configured *per guest*, not per pool, so a
  container created on `AppData` without a job is a single point of failure
  that looks identical in the storage view (hence Terraform owning the job
  rather than a hand-run `pvesr`); and it's asynchronous, so a failover loses
  up to `replication_schedule` — five minutes — of commits. It is not a
  backup; `pve_backup`'s vzdump is.
- `full_page_writes = off` on the Proxmox instance, and only there. It guards
  against a torn page, which copy-on-write storage cannot produce, so on ZFS
  the full-page images are pure write amplification that also becomes
  replication traffic. The same setting on ext4 or xfs risks unrecoverable
  corruption — which is why it's in `ansible/proxmox/vars.yml` and must never
  migrate to `common/vars.yml`.
- Terraform provisions the guest and stops there — it has no way to deploy an
  app into an LXC, so Ansible picks up from the point the container exists.
- Proxmox guests keep all their state on their own rootfs; no bind mounts, no
  NFS shares mounted into a container. Counter-intuitive if the goal is "back
  the repos up to the NAS", but `vzdump` excludes bind mounts by design, so a
  bind-mounted repo directory is the one thing *missing* from the archive.
  Everything local means one archive is a complete filesystem restore point.
  Getting a copy off-box is the node's job (`pve_backup` → NFS storage on the
  NAS + a vzdump schedule), not the container's.
- State an app keeps *outside* its container is that app's problem. Forgejo's
  database is on the shared Postgres LXC — a different guest, and so a
  different vzdump archive — so a systemd timer in the
  container `pg_dump`s it onto the container's own disk, and the vzdump archive
  carries both halves. The dump schedule and the vzdump window are a pair —
  move one, move the other.
- Proxmox hosts come from the `community.proxmox` dynamic inventory plugin
  querying the Proxmox API, so `terraform apply` creating an LXC doesn't also
  require an inventory edit.
- The `terraform` tag is what marks a Proxmox guest as this repo's to manage.
  The dynamic inventory returns every LXC on the cluster, hand-made ones
  included, so `inventory/proxmox.yml` filters on that tag — both Terraform
  modules set it (`tags` in their `variables.tf`), nothing else does. It
  filters rather than narrowing the `proxmox_guests` condition, so an
  unmanaged guest is absent from `all` as well and no future play can reach
  it by accident. The failure mode is silent in one direction: a new module
  that forgets the tag creates a container Ansible simply never visits, with
  no error anywhere — same shape as an omitted `stacks:` entry, and the first
  thing to check if a freshly-applied guest is never deployed to.
- Terraform state lives in Postgres (`backend "pg"`), not a local file, so CI
  and a workstation see the same state — specifically the **CloudNativePG
  cluster on k3s**, not the shared Postgres this configuration provisions on
  Proxmox. That would be circular: Terraform would need the database to exist
  in order to create the container the database runs in. State goes somewhere
  Terraform has no hand in building, which breaks the cycle outright at the
  cost of a dependency the rest of the repo avoids — `terraform apply` now
  needs the Pis up. Worth it over the alternative (local state, apply,
  `init -migrate-state`), which has to be got right exactly once. Bootstrap
  ordering is `docs/postgres-proxmox.md`.
- Every app keeps the same `common/` + `ansible/` + `terraform/` shape even
  when only one platform is in use, so adding the other later doesn't require
  restructuring.
- Removal on Unraid is explicit (`state: absent` on the stack entry), never
  implied by deleting the entry. Ansible stores no record of the last run, so
  an omitted stack isn't removed — it's just never visited again, and keeps
  running unmanaged. The entry stays as a tombstone documenting that the app
  was deliberately removed. Volumes, images and the app's database are
  further opt-ins (`remove_volumes`, `remove_images`, `remove_database`)
  rather than being implied by `state: absent`, and the Vault path is never
  touched — a default teardown is meant to be reversible. `lxc_app` has no
  counterpart yet; it would need a per-app `uninstall.yml`.
- The k3s cluster's join token is a fixed value from Vault
  (`homelab/k3s-homelab-utils` → `K3S_TOKEN`), not the one `k3s server`
  generates for itself on first install. Both the control-plane play and the
  worker play look it up independently, so a full rebuild — wipe both SD
  cards, reinstall — reproduces the same cluster identity instead of needing
  a freshly-generated token fetched off the server and redistributed by hand.
  Same "generate once, store it, never let a redeploy invent a new one"
  reasoning as Forgejo's fixed secrets, applied to cluster identity instead
  of app state.
- The SSH key the k3s plays connect with is fetched from Vault
  (`homelab/ci/ssh-k3s`) by `playbooks/k3s_ssh_key.yml` and written to
  `local/`, which both `k3s.yml` and `k3s_maintenance.yml` import as their
  first play — rather than the operator loading it into `ssh-agent` first, as
  the Unraid manual flow still expects. Same arrangement CI already uses for
  the Unraid key, just moved into the playbook because these two are
  manual-only and there's no workflow to hold the fetch step. It's a separate
  playbook, not a `pre_tasks` block, because both entry points need it and
  neither can assume the other ran. It targets `k3s_cluster` rather than
  `localhost` for a non-obvious reason: the implicit localhost isn't a member
  of `all`, so it wouldn't inherit `vault_addr`/`vault_kv_mount`/
  `vault_auth_method` from `group_vars/all.yml`. Nothing in it connects to a
  Pi — `gather_facts: false` plus `delegate_to: localhost`, necessarily, since
  the key isn't on disk yet at that point. Falling back to a local key takes
  *two* overrides (`-e k3s_ssh_key_fetch=false` alongside
  `-e ansible_ssh_private_key_file=...`): pointing at another key alone
  doesn't help if Vault is what's broken, because the fetch play fails before
  any other play runs.
- `playbooks/k3s.yml` is deliberately **not** wired into `deploy.yml` or
  `.forgejo/workflows/deploy.yml`. Converging 4 physical nodes on every push
  is a bigger blast radius than restarting a Compose stack — the same
  reasoning that keeps `terraform apply` manual for Proxmox. Run it by hand.
- `playbooks/k3s.yml`'s plays are tagged so one piece can be converged on its
  own — `nodes`, `metallb`, `monitoring`, `postgres`, `services` (those three
  cluster services together), `upgrades`, `kubeconfig`. Play-level tags, one
  per play, not per-task tags inside the roles: the plays already are the
  units, and tagging tasks would mean every role carrying tags for a
  selectivity only this playbook needs. This is the k3s equivalent of
  `only_stacks` for Unraid — a cluster service isn't a `stacks:` entry, so
  `only_stacks` can't reach it. The SSH-key import play is tagged `always`
  rather than getting a name of its own, since it's the prerequisite for
  anything else connecting and has to survive every `--tags` filter. Note
  what a tagged service run does *not* cover: the roles only drop manifests
  for k3s's controllers to reconcile, so a green playbook means the files
  landed, not that the workload came up.
- `k3s_node` is one role, not two, even though a server and an agent install
  differently. Both need the same OS prep (cgroups, swap), and which install
  path runs is a group var (`k3s_node_role: server`/`agent`, set per
  inventory group) rather than a second role — so the shared prep can't drift
  between the two nodes types the way copy-pasted tasks eventually would.
- Extra `INSTALL_K3S_EXEC` flags split across two vars, not one:
  `k3s_extra_args` (both roles) and `k3s_server_extra_args` (control-plane
  only). `k3s agent` doesn't understand server-only flags like `--disable`
  and fails to start if handed one, so a flag like `--disable=servicelb`
  (which `k3s_metallb` needs to avoid fighting k3s's bundled ServiceLB for
  the same IPs) can't safely live in the shared list.
- `k3s_node`'s install tasks track the exec line they last installed a node
  with (a small marker file next to k3s's own config) and reinstall —
  restarting the `k3s`/`k3s-agent` service, not the node — whenever it
  drifts from what `k3s_extra_args`/`k3s_server_extra_args`/`k3s_api_tls_san`
  currently say, even if `k3s_version` hasn't changed. Deliberately folded
  into the ordinary install task rather than a separate maintenance
  playbook: `playbooks/k3s.yml` is already manual-only and documented as
  safe to re-run, and unlike the reboots `k3s_maintenance` guards, bouncing
  the k3s process doesn't take pods down — containerd keeps them running
  underneath it. The worker play still runs with `serial: 1` so at most one
  node's kubelet is ever mid-restart at once.
- `k3s server`/`k3s agent` are installed with an explicit `--node-name
  {{ inventory_hostname }}`, not left to default to the OS hostname. This is
  what lets `roles/k3s_maintenance` address a node by
  `inventory_hostname` when draining/uncordoning — the k8s node object and
  the Ansible host are guaranteed to be the same string.
- Patching a k3s Pi is split into two roles that don't know about each other
  directly, only through a file: `unattended_upgrades` installs updates
  hands-off but with `Automatic-Reboot "false"`, and `k3s_maintenance` is the
  only thing that actually reboots a node — triggered by the presence of
  `/var/run/reboot-required`, which is the OS's own signal, not something
  either role invents. Splitting it this way means the risky half (taking a
  node out of the cluster) is exactly one thing (`playbooks/k3s_maintenance.yml`,
  `serial: 1`), not entangled with the routine half (installing packages),
  which runs unattended every day on every node without anyone thinking about
  it.
- `k3s_maintenance`'s kubectl calls (`drain`, `wait`, `uncordon`) are
  delegated to the control-plane host and run as `k3s kubectl`, k3s's own
  bundled client, rather than requiring a kubectl install or a kubeconfig
  anywhere else — including on whatever eventually runs this on a schedule.
  This also has to work when the node currently being processed *is* the
  control plane: delegating a host to itself is just a normal SSH connection,
  used right up until the moment that connection reboots out from under it.
- `k3s_monitoring` (a lean kube-prometheus-stack, for OpenLens and similar
  tools) is installed as a `HelmChart` custom resource dropped into k3s's own
  auto-deploying manifests directory, not run through a `helm` binary or the
  `kubernetes.core.helm` collection — k3s ships a helm-controller that
  reconciles anything found there, the same mechanism it uses to install its
  own bundled Traefik and ServiceLB. That keeps the pattern this repo already
  uses everywhere else (Ansible renders a file, something else converges on
  it) instead of adding a second, unrelated way to reach the cluster from the
  controller. Grafana and Alertmanager are left disabled — OpenLens brings
  its own dashboards and this cluster doesn't page anyone — and the
  control-plane scrape targets (`kubeControllerManager`, `kubeScheduler`,
  `kubeProxy`, `kubeEtcd`) are disabled too, since k3s bundles those into one
  static binary instead of exposing them the way the chart expects; leaving
  them on just produces permanently-"down" targets, not a working scrape.
- `k3s_metallb` (MetalLB, L2 mode) follows the same `HelmChart` CR pattern as
  `k3s_monitoring`, plus a second, plain manifest for its
  `IPAddressPool`/`L2Advertisement` config dropped in the same
  auto-deploying directory — k3s's deploy controller applies any manifest it
  finds there, not only `HelmChart` CRs, and retries one referencing CRDs
  that don't exist yet rather than failing outright, so the config doesn't
  need to wait on the chart install finishing first. It replaces k3s's
  bundled ServiceLB rather than running next to it — both would otherwise
  hand out IPs for the same `LoadBalancer` Services — so
  `k3s_server_extra_args` in `inventory/group_vars/k3s_cluster.yml`
  carries `--disable=servicelb` (server-only — `k3s_extra_args`, applied to
  both server and agent, can't carry a server-only flag like `--disable`
  without breaking agent installs). The IP pool (`k3s_metallb_address_range`, same
  file) has no built-in default; the role fails fast rather than silently
  advertising an empty pool.
- `k3s_postgres` (CloudNativePG) is the k3s cluster's own shared Postgres —
  a third platform for `src/shared/postgres/`, alongside Unraid and Proxmox,
  deployed via Helm rather than Compose or a native install. Chosen over
  Bitnami's postgresql-ha (repmgr+pgpool, and Bitnami's free chart/image
  catalog was restructured into a "legacy" repo in 2025) and the Zalando
  operator (Patroni-based, heavier) because it fits the same `HelmChart` CR
  pattern already established: the operator installs as a chart CR, same
  shape as `k3s_metallb`/`k3s_monitoring`, and the actual cluster is a plain
  `Cluster` CR manifest — same "config manifest that outlives the CRDs it
  references" trick as `k3s_metallb`'s `IPAddressPool`. It's a separate
  physical instance from the Docker-based shared/postgres (pinned to 13),
  but reuses the same Vault path (`homelab/shared/postgres`) rather than a
  k3s-specific one — one superuser identity for the "shared postgres"
  concept regardless of which platform it's running on, consistent with how
  every host already draws from that same path. 1 primary + 1 replica,
  scheduled on worker Pis only via `nodeAffinity` — the control plane stays
  free of app pods, same boundary `k3s_metallb`/`k3s_monitoring` already
  keep. HA replication only for now, no backups.
- The CNPG cluster is published on the LAN through MetalLB at a pinned
  `k3s_postgres_loadbalancer_ip`, on top of the three `ClusterIP` Services
  CNPG makes for every `Cluster`. Declared inside the `Cluster` CR as a
  managed service (`.spec.managed.services.additional`, `selectorType: rw`)
  rather than as a Service manifest of this repo's own, so the operator owns
  the selector and the address follows a failover instead of needing to be
  re-pointed by hand. Pinned rather than auto-assigned because the address is
  meant to be written down in config elsewhere; it comes out of the bottom of
  `k3s_metallb_address_range` and would need a second `autoAssign: false`
  pool if anything else on this cluster ever wants a fixed IP. This is what
  closes the reachability half of per-app DB provisioning — the
  `community.postgresql` tasks `compose_stack`/`lxc_app` use can now reach
  this instance the same way they reach the Unraid one. `roles/k3s_app` is
  the other half, and uses exactly those tasks; note the asymmetry it
  relies on, which is easy to misread as a mistake. The *provisioning* runs
  from the Ansible controller and therefore uses the LoadBalancer address
  (`k3s_postgres_loadbalancer_ip`), because `delegate_to: localhost` is off
  the cluster and can't route to a ClusterIP. The *app* uses the in-cluster
  `shared-postgres-rw` DNS name, because sending pod traffic out to the LAN
  and back would put MetalLB's L2 speaker in the path of every query for no
  benefit. Two addresses for one database, each correct for its caller.
- `roles/k3s_app` is the cluster's counterpart to `compose_stack` and
  `lxc_app`: the generic role that deploys an *app*, driven by `k3s_apps:`
  in `group_vars/k3s_cluster.yml`, as distinct from the `k3s_*` roles that
  each deploy one cluster service. It renders whatever `*.yaml.j2` the app
  ships under `src/<app>/ansible/kubernetes/` into k3s's auto-deploying
  manifests directory, so adding a manifest to an app is dropping a file
  next to the others rather than a role change. Secrets go into a separate
  Kubernetes Secret rendered from Vault, never interpolated into the
  manifests — the same committed-config/generated-secrets split as Unraid's
  static compose file plus rendered `.env`, which is why the manifests can
  stay 0644 on the node while the Secret is 0600.
- `k3s_apps:` lives in `group_vars/k3s_cluster.yml`, not `host_vars/`, unlike
  both other platforms. An app is deployed to the cluster, not to a node:
  the play runs against `k3s_control_plane` only because that's where the
  manifests directory is, which is an implementation detail of how k3s is
  reached rather than a statement about where the app runs.
- Removal on k3s is one pass, not two. `compose_stack` needs the compose file
  still on disk to run `docker compose down`, so `state: absent` and deleting
  `src/<app>/` have to be separate commits (see "Removing an app" below).
  k3s's deploy controller tracks which resources each manifest file created,
  so deleting the file *is* the teardown and `src/<app>/` can go in the same
  commit. The tombstone convention still applies for the same reason as
  everywhere else — an entry dropped from `k3s_apps:` is never visited again
  and keeps running unmanaged.
- `k3s_cert_manager` is the fourth cluster service and the first that exists
  purely for apps rather than for the cluster — nothing in the cluster needs
  a certificate, `k3s_app`'s tenants do. It's still a cluster service rather
  than an app, on the same reasoning as the shared Postgres: one
  `ClusterIssuer` that every app's Ingress annotates itself against, rather
  than each app carrying its own ACME account and DNS credentials. DNS-01,
  not HTTP-01, because HTTP-01 needs Let's Encrypt to reach the cluster from
  the internet on port 80 and this LAN deliberately isn't reachable — which
  has the useful side effect that a certificate can be issued *before* DNS
  points at the cluster, so a migration's TLS is settled before its cutover.
  The solver stanza itself has no default and the role fails fast without
  one: it depends on who runs the DNS, which the repo can't know. Same
  fail-fast-on-unset treatment as `k3s_metallb_address_range`.
- Apps on k3s reach the outside through the Traefik k3s already bundles,
  on the MetalLB address its Service holds, rather than each getting its own
  LoadBalancer IP. One entrypoint and host-based routing means adding an app
  costs a DNS record, not a pool address — the opposite of the choice made
  for `k3s_postgres`, which is pinned to its own IP precisely because
  Postgres isn't HTTP and can't be name-routed.
- `k3s_traefik` is the one cluster-service role that installs nothing: k3s
  installs Traefik itself, so the role only adjusts it, via a
  `HelmChartConfig` merged over k3s's own `HelmChart` rather than an edit to
  the `traefik.yaml` k3s rewrites on every server start. Its scope is the
  dashboard, which a stock k3s 404s — Traefik still builds it, but the chart
  stopped shipping the router that reaches it in v28. The role restores that
  router on Traefik's internal `traefik` entrypoint (port 9000, unpublished →
  port-forward only), and optionally publishes the dashboard on a hostname
  behind an Authentik forward-auth middleware when
  `k3s_traefik_dashboard_host` is set. Three things worth keeping straight:
  the internal route stays on even when the hostname one exists, because
  authenticated access depends on Authentik → CNPG → a healthy cluster,
  exactly what you'd open the dashboard to diagnose; the published route is
  an `IngressRoute` with an explicit cert-manager `Certificate` rather than
  an annotated `Ingress`, because the dashboard is `api@internal` and has no
  Service to point an `Ingress` at; and the role fails fast if a hostname is
  set without an auth address, since the failure mode is publishing every
  router, service and middleware on the cluster to the LAN unauthenticated.
  Authentik's provider must be **forward auth (domain level)** —
  single-application mode needs `/outpost.goauthentik.io/` routed on the
  dashboard's own host, a cross-namespace service reference Traefik rejects
  unless `allowCrossNamespace` is on.

## Adding a new app

1. `src/<app>/common/vars.yml` — `env_defaults:` for the portable config.
2. Whichever platform(s) it targets:
   - **Unraid:** `src/<app>/ansible/unraid/` with `docker-compose.yml`
     (static, `${VAR}`-driven), `vars.yml` of overrides, `.env.example`
     documenting every var.
   - **Proxmox:** `src/<app>/ansible/proxmox/` with `vars.yml`,
     `install.yml`, and config/systemd templates; plus
     `src/<app>/terraform/` defining its LXC, wired into
     `build/config/terraform/main.tf`.
   - **k3s:** `src/<app>/ansible/kubernetes/` with `vars.yml` and at least
     one `*.yaml.j2` manifest — normally a `HelmChart` CR, plus anything the
     chart won't create for you. See `src/authentik/` for a worked example,
     including the two things a chart usually leaves out: a PVC, and secrets
     referenced by `secretKeyRef` rather than set as values.
3. Populate its Vault path (`homelab/<app>`) with real secret values.
4. Declare it on the target:
   - Unraid → `host_vars/<host>.yml` `stacks:` list.
   - Proxmox → `host_vars/<guest>.yml` `apps:` list.
   - k3s → `group_vars/k3s_cluster.yml` `k3s_apps:` list.
   All three take the same entry shape (name, src, vault_path, optional
   `db:`, optional `state:`). Never declare the same app on more than one at
   once — they share a Vault path and a database, so two live deployments
   corrupt each other's state. Moving an app *between* platforms is
   therefore a cutover with a verification step in the middle, not an edit;
   `docs/authentik-migration.md` is the worked example.
5. Leave the unused platforms' folders as README placeholders rather than
   deleting them — the shape is the point.

## Removing an app

**Unraid.** Set `state: absent` on its `stacks:` entry and run
`playbooks/unraid.yml` — don't delete the entry, which removes nothing. Then
delete `src/<app>/` in a *second* pass, never before: `docker compose down`
needs the compose file to know what it's tearing down. Full detail, including
the opt-in flags for volumes/images/database, is in README.md → "Removing an
app".

**k3s.** Same `state: absent` tombstone rule, but one pass — deleting the
manifests is the teardown, because k3s's deploy controller garbage-collects
what each file created, so `src/<app>/` can go in the same commit. The
database, PVCs and Vault path deliberately survive; there's no
`remove_volumes`/`remove_database` equivalent yet, and adding one is a
"when it's actually wanted" job rather than a guess at the shape.

**Proxmox.** No teardown path at all — `lxc_app` would need a per-app
`uninstall.yml`, and neither app that ships an `install.yml` has one. Note
this is worse for the shared Postgres than for an ordinary app: tearing it
down means every other app's database as well.

## K3s cluster (homelab-utils)

4 Raspberry Pis, `inventory/hosts.yml` → `k3s_cluster` (`k3s_control_plane`:
1 host, `k3s_workers`: 3). No Terraform — the Pis already exist. The cluster
itself has no `src/<app>/` entry (it isn't an app), but apps now land on it:
`k3s_apps:` in `group_vars/k3s_cluster.yml`, deployed by `roles/k3s_app` from
`src/<app>/ansible/kubernetes/`. `playbooks/k3s.yml` is manual-only (see "Key
decisions" above); run it with:

```sh
cd build/config/ansible
ansible-playbook playbooks/k3s.yml

# or one piece of it — see "Key decisions" above for the tag list
ansible-playbook playbooks/k3s.yml --tags postgres

# apps only, or one app
ansible-playbook playbooks/k3s.yml --tags apps
ansible-playbook playbooks/k3s.yml --tags apps -e only_apps=authentik
```

`--tags services` covers the cluster's own infrastructure and `--tags apps`
what runs on top of it; neither implies the other, so "everything except
reinstalling k3s" is `--tags services,apps`.

Prerequisites, once, before the first run: the `ansible` user exists on every
Pi with NOPASSWD sudo and the **public** half of this repo's k3s SSH key
installed, and `homelab/k3s-homelab-utils` → `K3S_TOKEN` plus
`homelab/ci/ssh-k3s` → `PRIVATE_KEY` are set in Vault. Nothing needs setting
up on the controller beyond `VAULT_ADDR`/`VAULT_TOKEN` — the private key comes
from Vault at run time (see "Key decisions" above). Full detail in README.md →
"K3s (Raspberry Pi)" and `docs/vault-secrets.md`.

**Rebuilding**: reimage the SD card(s), redo the `ansible` user + SSH key
step, then re-run the playbook — the fixed Vault token means the rebuilt
node(s) rejoin the same cluster identity rather than needing a token hunted
down from a live server. **Adding a fifth Pi**: add it under
`k3s_control_plane` or `k3s_workers` in `hosts.yml`; no `host_vars/` entry
needed, `k3s_node_role` comes from the group. This repo's `k3s_node/server.yml`
assumes a single control-plane node — adding a second one for HA needs
changes there, not just an inventory edit.

**Patching**: `playbooks/k3s.yml` also applies `roles/unattended_upgrades` to
every node, so updates install themselves daily with no login required — but
never reboot themselves (`Automatic-Reboot "false"`). Run
`playbooks/k3s_maintenance.yml` to actually apply a pending reboot: it checks
`/var/run/reboot-required` per node and, only where set, drains → reboots →
waits for `Ready` → uncordons, `serial: 1` so only one node is ever down.
Manual-only for now, same as `playbooks/k3s.yml` — meant to eventually run on
a schedule from a self-hosted Forgejo Actions runner rather than by hand.

**Metrics**: `playbooks/k3s.yml` also applies `roles/k3s_monitoring`, a lean
kube-prometheus-stack (no Grafana/Alertmanager) so OpenLens and similar tools
can show node/pod metrics — see "Key decisions" above for why it's a
`HelmChart` CR rather than a `helm` install, and README.md → "Metrics
(Prometheus)" for pointing OpenLens at it.

**LoadBalancer IPs**: `playbooks/k3s.yml` also applies `roles/k3s_metallb`,
MetalLB in L2 mode, so `type: LoadBalancer` Services get real LAN IPs instead
of staying `ClusterIP`-only. Set the pool it hands out from —
`k3s_metallb_address_range` in `inventory/group_vars/k3s_cluster.yml` — to a
range your DHCP scope and every static assignment in `hosts.yml` steer clear
of before the first run; see "Key decisions" above for why it replaces
ServiceLB rather than running alongside it, and README.md → "LoadBalancer
IPs (MetalLB)" for more.

There's no *node* teardown path, unlike `state: absent` for apps — removing a
Pi today means wiping its SD card and deleting its inventory entry by hand.

**Shared Postgres**: `playbooks/k3s.yml` also applies `roles/k3s_postgres`,
CloudNativePG — the cluster's own shared Postgres, a third platform for
`src/shared/postgres/` alongside Unraid and Proxmox. See "Key decisions"
above for why CloudNativePG and the topology, and README.md → "Postgres
(CloudNativePG)" for connecting to it. Per-app database provisioning works
now (`roles/k3s_app`, `db:` on a `k3s_apps:` entry) — note the two-address
asymmetry described in "Key decisions". Still **no backups**: HA replication
only, which protects against a dead SD card and not against a dropped table.
That gap matters more now that Authentik's database lives here rather than
inside whatever covers Unraid's `/mnt/user/appdata`; a CNPG `ScheduledBackup`
is the obvious next piece of work.

**TLS**: `playbooks/k3s.yml` also applies `roles/k3s_cert_manager` —
cert-manager plus one `ClusterIssuer` that every app's Ingress annotates
itself against. Two values have no default and the role refuses to run
without them: `k3s_cert_manager_acme_email` and `k3s_cert_manager_solver`
(the DNS-01 stanza, which depends on your DNS provider — a commented
Cloudflare example is in `group_vars/k3s_cluster.yml`). Use the Let's Encrypt
staging directory while working a solver out; production allows five failed
validations per hostname per hour.

**Traefik dashboard**: `playbooks/k3s.yml` also applies `roles/k3s_traefik` —
the only cluster-service role that installs nothing, since k3s installs
Traefik itself. Out of the box the dashboard is reachable only by
port-forward (`kubectl -n kube-system port-forward deploy/traefik 9000:9000`,
then `http://127.0.0.1:9000/dashboard/` — trailing slash required; without
this role a stock k3s 404s there). Publishing it on a hostname behind
Authentik is one commented-out line, `k3s_traefik_dashboard_host` in
`group_vars/k3s_cluster.yml`, plus a domain-level forward-auth Proxy Provider
on the Authentik side. See "Key decisions" above and README.md → "Dashboard
(Traefik)".

**Apps**: `k3s_apps:` in `group_vars/k3s_cluster.yml`, deployed by
`roles/k3s_app`. Authentik is the first and currently only one — it moved off
a hand-made Unraid container rather than being deployed fresh, so
`src/authentik/ansible/unraid/` is a README explaining why there's no compose
file there. The cutover procedure is `docs/authentik-migration.md`; the part
worth knowing without reading it is that the deployed `AUTHENTIK_VERSION`
must match the version a restored database was dumped from, because
Authentik's migrations run on startup and are one-way.