homelab/README.md

# Homelab IaC

Deploy apps as code across two platforms: Unraid server(s), where apps run as
Docker Compose stacks, and Proxmox, where Terraform creates an LXC and the
app is installed into it natively. Ansible is the deployment tool on both.
Secrets are pulled from HashiCorp Vault at deploy time; CI/CD is Forgejo
Actions once Forgejo itself is running.

There's also a third piece of infrastructure alongside those two: a bare-metal
k3s cluster (`homelab-utils`) across 4 Raspberry Pis, bootstrapped by Ansible
too. It doesn't fit the `src/<app>/` table above the way Unraid/Proxmox apps
do — Terraform never provisions it, and no `stacks:`/`apps:` list drives
what's on it — so it gets its own inventory group and role instead. It does
run shared services deployed via Helm, though: `k3s_monitoring`, `k3s_metallb`,
and the cluster's own shared Postgres (`k3s_postgres`, a third platform for
`src/shared/postgres/`). See `playbooks/k3s.yml` and "K3s (Raspberry Pi)"
below.

See `CLAUDE.md` for the short version of the architecture decisions.

## Layout

```
src/
  <app>/
    common/vars.yml        config that holds on either platform
    ansible/
      unraid/              docker-compose.yml + .env.example + platform overrides
                           (+ optional icon.png for the Unraid Docker page)
      proxmox/             install.yml + config/systemd templates + platform overrides
      kubernetes/          vars.yml + *.yaml.j2 manifests (normally a HelmChart CR)
    terraform/             this app's LXC definition (Proxmox only)
  shared/<service>/        same shape, for services multiple apps use
build/config/
  ansible/
    inventory/
      hosts.yml            static: unraid_servers, proxmox_nodes, k3s_cluster
      proxmox.yml          dynamic: proxmox_guests, from the Proxmox API
      group_vars/all.yml   Vault connection, Compose Manager project path
      group_vars/k3s_*.yml k3s cluster name/version, per-role vars (server/agent)
      host_vars/<host>.yml which stacks/apps deploy there
                           (k3s apps are in group_vars/k3s_cluster.yml instead)
    roles/compose_stack/   Unraid: sync compose + render .env + provision DB + up
                           (deploy.yml / remove.yml, picked by the entry's state:)
    roles/lxc_app/         Proxmox: install natively + render config + systemd
    roles/k3s_app/         k3s: render app manifests + Secret + provision DB
                           (deploy.yml / remove.yml, picked by the entry's state:)
    roles/pve_backup/      Proxmox node: NAS backup storage + vzdump schedule
    roles/k3s_node/        k3s Pi: cgroups/swap prep + install server or agent
    roles/k3s_monitoring/  k3s control plane: lean kube-prometheus-stack via HelmChart CR
    roles/k3s_metallb/     k3s control plane: MetalLB (LoadBalancer IPs) via HelmChart CR
    roles/k3s_postgres/    k3s control plane: shared Postgres (CloudNativePG) via HelmChart CR
    roles/k3s_cert_manager/ k3s control plane: cert-manager + ClusterIssuer via HelmChart CR
    roles/unattended_upgrades/  any apt host: hands-off patching, no auto-reboot
    roles/k3s_maintenance/ k3s Pi: drain + reboot-if-required + uncordon, one at a time
    playbooks/
      deploy.yml           everything (Unraid + Proxmox — not k3s, see below)
      unraid.yml           just Unraid
      proxmox.yml          just the Proxmox guests
      pve_host.yml         just the Proxmox node (backups)
      k3s.yml               bootstraps the k3s cluster (manual-only, not in deploy.yml)
      k3s_maintenance.yml   rolls pending reboots across it, serial: 1 (manual-only)
  terraform/               Proxmox provider, pg state backend, LXC module calls
.forgejo/workflows/      CI/CD (needs a self-hosted runner, see below)
docs/vault-secrets.md    Vault path/key reference
docs/authentik-migration.md  hand-run cutover: Authentik from Unraid to k3s
```

Terraform's scope stops at the guest — it creates the LXC and nothing more,
because the Proxmox provider has no way to install an application. Ansible
picks up from there, which is why it's the common tool and Terraform is
Proxmox-only.

## How config is layered

An app's settings are split by whether they survive a change of platform:

- `src/<app>/common/vars.yml` — version, ports, domain, database name/user.
- `src/<app>/ansible/<platform>/vars.yml` — the rest. Data paths differ
  (`/mnt/user/appdata/forgejo` vs `/var/lib/forgejo`), and so does how the
  database is reached (the `shared-postgres` Docker network alias only exists
  on Unraid).

Both roles merge them the same way: common, then platform overrides, then
Vault secrets last. Nothing is restated between platforms.

## How a deploy works

**Unraid.** For each stack in a host's `stacks:` list, `compose_stack`:

1. Copies `src/<stack>/ansible/unraid/docker-compose.yml` to the host's
   Compose Manager project folder, unmodified — it's static and only
   references `${VAR}`.
2. Merges the two `vars.yml` layers with the stack's Vault KV path and
   renders the project's `.env`. Nothing secret ever touches git.
3. If the stack declares a `db:` block, ensures its database/role exist on
   the shared Postgres instance (idempotent — safe to add new apps to the
   shared service over time without touching existing data).
4. Copies `icon.png`, if the stack ships one, to both the project folder and
   `/mnt/user/appdata/icons/` (see "Icons on the Unraid Docker page" below).
5. Runs `docker compose up -d` for the stack.

**Proxmox.** `terraform apply` creates the LXC first. Then, for each app in a
guest's `apps:` list, `lxc_app` does the same config merge and database
provisioning, runs that app's `ansible/proxmox/install.yml`, and manages its
systemd unit instead of a Compose project.

**k3s.** For each app in `k3s_apps:` (`group_vars/k3s_cluster.yml`, not
host_vars — an app belongs to the cluster, not a node), `k3s_app`:

1. Does the same two-layer config merge, using
   `src/<app>/ansible/kubernetes/vars.yml` as the platform layer.
2. Provisions the app's database/role on the CloudNativePG cluster if it
   declares a `db:` block — the same `community.postgresql` tasks the other
   two platforms use, pointed at MetalLB's LoadBalancer address for the CNPG
   primary.
3. Renders the app's Vault path into a Kubernetes `Secret` (plus the
   `Namespace`), keys passed through verbatim.
4. Renders every `*.yaml.j2` the app ships into
   `/var/lib/rancher/k3s/server/manifests/`, where k3s's own deploy and helm
   controllers pick them up.

Nothing here talks to the Kubernetes API — no `helm`, no kubeconfig, no
`kubernetes.core`. Ansible writes files; k3s reconciles them. The consequence
worth remembering: **a green run means the manifests landed, not that the
workload came up.** Check with `kubectl -n <namespace> get pods`.

Secrets land on disk in the rendered `.env` (root-readable, on the Unraid
box) or, on k3s, in a 0600 manifest and then in etcd as an ordinary
Kubernetes Secret — this is "fetch from Vault at deploy time," not a
zero-secrets-at-rest model. Fine for a homelab; revisit if that changes.

## Icons on the Unraid Docker page

There are two icons per stack, and they come from completely different places.
Commit **one** `src/<app>/ansible/unraid/icon.png` and the deploy feeds both.
A stack running more than one container can commit
`icon-<service>.png` alongside it for the extra services — see "More than one
container" below.

### The stack row

Compose Manager draws the collapsible header row for the whole project, and
serves its icon straight off disk from `icon.png` in the project folder —
`/boot/config/plugins/compose.manager/projects/<stack>/icon.png`, the same
directory the compose file and rendered `.env` land in. No label, no
template, no URL: the file is there or the row has no icon. `.jpg`, `.gif`,
`.svg` and an extensionless `icon` also work, since the plugin serves the file
itself rather than handing a path to the webgui — but stick to PNG so the same
file can serve the labels below.

This is a feature of **Compose Manager Plus** (`mstrhakr/compose_plugin`), the
maintained fork. The original Docker Compose Manager plugin (`dcflachs`,
archived April 2026) has no icon support at all — on that one the copied file
is simply ignored, so the deploy is safe either way, you just won't see it.

### The containers under it

Unraid's Docker page gets each *container's* icon, WebUI link and console
shell out of the dockerMan template that created it, under
`/boot/config/plugins/dockerMan/templates-user/`. A Compose stack never
creates one, which is why compose-deployed containers show a question mark and
no WebUI entry. Unraid 6.10+ falls back to Docker labels when there's no
template, so each service sets them itself:

```yaml
labels:
  net.unraid.docker.icon: ${STACK_ICON}
  net.unraid.docker.webui: "http://[IP]:${FORGEJO_HTTP_PORT}/"
  net.unraid.docker.shell: sh
```

They're per **service**, not per stack — a stack with two containers labels
each one separately, and can point them at different icons.

- **icon** — `${STACK_ICON}` is set by the role, not by hand. The same
  committed `icon.png` is copied a second time to
  `/mnt/user/appdata/icons/<stack>.png` and the label points there; ship no
  `icon.png` and the variable renders empty, leaving the placeholder. To use a
  hosted image instead, set `STACK_ICON` in the app's
  `ansible/unraid/vars.yml` — it's merged over the role's value.

  A second copy rather than reusing the project-folder one because this label
  is a path the *webgui* resolves on every Docker page load, and that path
  shouldn't run through the flash drive.

  **PNG only here** — unlike the stack row, this goes through the webgui's own
  icon handling, where SVG renders as the question-mark fallback and WebP
  renders nothing at all. A remote URL is fetched when the page renders, so an
  unreachable host breaks the icon — the reason a committed file is the
  default.

- **webui** — `[IP]` is substituted with the host's address. The port must be
  the *published* one, so interpolate the stack's own port variable rather
  than the container port. Omit the label entirely for something with no web
  interface (as `shared/postgres` does) — an empty value still draws a WebUI
  entry that goes nowhere.

- **shell** — `sh` or `bash`, whichever the image actually has. Alpine-based
  images (Forgejo) need `sh`; Debian-based ones (Postgres, Shelfarr) can take
  `bash`.

One caveat: editing a compose container's labels through the Unraid UI writes
a template for it, and the template then wins over the labels until you clear
it. Change these in the compose file and redeploy, not in the webgui.

### More than one container

`${STACK_ICON}` is a single value — the stack's `icon.png` — so a stack with
two services would otherwise have to give both the same image. Since the
labels are per-service, commit `icon-<service>.png` next to `icon.png` for
each additional container, named for the **compose service**:

```
src/arr/ansible/unraid/
  icon.png              → ${STACK_ICON}           → /mnt/user/appdata/icons/arr.png
  icon-prowlarr.png     → ${STACK_ICON_PROWLARR}  → /mnt/user/appdata/icons/arr-prowlarr.png
```

The variable is the service name uppercased with `-` folded to `_`
(`icon-shelfarr-libation.png` → `${STACK_ICON_SHELFARR_LIBATION}`). These get
the appdata copy only — there is one Compose Manager stack row and `icon.png`
already has it — and, like `STACK_ICON`, each can be overridden with a hosted
URL by setting the same name in the app's `ansible/unraid/vars.yml`.

The variable only exists when the file does. Interpolating
`${STACK_ICON_FOO}` with no `icon-foo.png` committed gets Compose's
unset-variable warning and the question-mark placeholder — so if the icon is
meant to be a URL, set it in `vars.yml` rather than relying on the file.

## Backups (Proxmox)

Proxmox guests keep **all** their state on their own rootfs. Nothing is
bind-mounted in from the NAS, which is the opposite of the obvious instinct —
the reason is that `vzdump` deliberately excludes bind mounts, so a
bind-mounted repository directory would be the one thing missing from the
backup that was made to protect it. Keeping everything local means one archive
is a complete copy of the guest's filesystem.

`playbooks/pve_host.yml` (role: `pve_backup`) configures the off-box copy on
the node: an NFS storage pointing at the NAS, and a scheduled `vzdump` job
writing to it. Retention is the storage's `prune-backups` settings. Point it at
a Proxmox Backup Server instead by setting `pve_backup_storage_type: pbs` in
`host_vars/<node>.yml`.

**Data that lives outside a container is that app's problem, not vzdump's.**
Forgejo's database is on the shared Postgres over on Unraid, so restoring its
container alone would give back every repository with no issues, pull requests,
users or permissions. The fix is a `pg_dump` on a systemd timer *inside* the
container, writing into its own filesystem — which puts the dump inside the
same archive as the repositories it belongs to. That only works if it finishes
before the backup window, so the two schedules are a pair:

| | Set in | Default |
|---|---|---|
| Forgejo database dump | `src/forgejo/ansible/proxmox/vars.yml` → `FORGEJO_DB_DUMP_ONCALENDAR` | 01:30 |
| vzdump job | `inventory/host_vars/pve.yml` → `pve_backup_schedule` | 02:00 |

Any future app with external state should follow the same pattern. Restore
steps are in `src/forgejo/ansible/proxmox/README.md`.

None of this covers the Unraid side, which has its own backup arrangements
outside this repo.

## Prerequisites

### Both platforms

- Ansible control node needs to be Linux — **on Windows, run it from WSL2**,
  not natively.
- If this repo is checked out on a Windows mount (e.g. under `/mnt/c/...`,
  which is where a Seafile-synced folder ends up) rather than the native WSL
  filesystem, Ansible will warn that `ansible.cfg` is being ignored because
  the directory looks world-writable — that's DrvFs reporting `777` on
  everything, not a real permissions problem. Either `export
  ANSIBLE_CONFIG="$(pwd)/ansible.cfg"` before running (from the
  `build/config/ansible/` dir), or fix it permanently by adding to
  `/etc/wsl.conf` inside WSL:
  ```ini
  [automount]
  options = "metadata,umask=22,fmask=11"
  ```
  then `wsl --shutdown` from PowerShell and reopen the shell.
- `pip install ansible hvac psycopg2-binary`
- `ansible-galaxy collection install -r build/config/ansible/requirements.yml`
- A reachable Vault instance, with secrets populated per
  `docs/vault-secrets.md`.

### Unraid

- Root SSH enabled on the Unraid box(es) (Settings → Management Access).
- Compose Manager (Docker Compose plugin) installed on Unraid — already
  done.
- **Python3 on each Unraid host itself**, not just the controller: the
  `compose_stack` role's tasks (`file`, `copy`, `template`,
  `docker_compose_v2`) run on the target, and Ansible modules — unlike
  `raw`/ad hoc shell — need a Python interpreter there to execute at all.
  Stock Unraid doesn't ship one. Installing it by hand into the running
  system doesn't survive a reboot, same reason as the SSH-key gotcha above —
  Unraid rebuilds its root filesystem from the flash drive on every boot.
  Install the **NerdTools** plugin (Community Applications → search
  "NerdTools", formerly "NerdPack") and enable `python3` in its package
  list — it reinstalls whatever you've selected on every boot itself, same
  mechanism Compose Manager already relies on. `ansible.cfg` already sets
  `interpreter_python = auto_silent`, so Ansible finds it automatically
  wherever NerdTools puts it; no path to hardcode. Verify after installing:
  ```sh
  ansible unraid_servers -m ping
  ```

### Proxmox

- Terraform CLI (>= 1.6) on whatever runs `terraform apply`.
- A Proxmox API token with rights to create containers, stored in Vault at
  `homelab/ci/proxmox` (see `docs/vault-secrets.md`). Both the Terraform
  provider and Ansible's dynamic inventory authenticate with it.
- A `terraform_state` database and `terraform` role on the **CloudNativePG
  cluster on k3s** (192.168.50.81), for `backend "pg"` — not on either shared
  Postgres this repo deploys, which would be circular. See "Bootstrapping"
  below.
- Container templates downloaded on the Proxmox node for whatever OS the
  LXCs are built from.
- An NFS export on the NAS for backup archives, reachable from the Proxmox
  node — set it in `inventory/host_vars/pve.yml`. See "Backups (Proxmox)".
- Root SSH from the Ansible controller to the Proxmox node itself, not just
  the guests: `playbooks/pve_host.yml` configures the node over SSH.
- Unlike Unraid, nothing special is needed *inside* the guests: they're
  ordinary Linux containers with Python already present, so Ansible works
  without the NerdTools workaround above.

### K3s (Raspberry Pi)

- 4 Raspberry Pis running Ubuntu Server (64-bit), already imaged and on the
  network at the addresses in `inventory/hosts.yml` → `k3s_cluster`.
- An `ansible` user on each Pi with **NOPASSWD sudo** and this repo's k3s SSH
  key installed as an authorized key — see "SSH access" below. Not automated
  by this repo; create it by hand (or via cloud-init at image time) before the
  first run of `playbooks/k3s.yml`.
- `homelab/k3s-homelab-utils` → `K3S_TOKEN` set in Vault *before* the first
  run — see `docs/vault-secrets.md`. Both the server and every agent read
  this same fixed value rather than one generating it and handing it to the
  other, which is what makes a rebuild reproducible.
- Nothing else special: `python3` ships with Ubuntu Server, and the k3s
  install script (`get.k3s.io`) handles containerd, the systemd unit, and
  everything else that isn't Pi-specific. The one Pi-specific thing —
  ensuring the memory cgroup controller is on — is handled by
  `roles/k3s_node` itself, not a prerequisite here.

#### Keeping it patched

`playbooks/k3s.yml` also applies `roles/unattended_upgrades` to every node —
apt updates install themselves on their own daily schedule, no login
required. It sets `Unattended-Upgrade::Automatic-Reboot "false"` though, so a
kernel or containerd update that needs a reboot to take effect just sits
applied-but-inactive until one happens; blindly auto-rebooting a k3s node
takes its pods down with no warning.

`playbooks/k3s_maintenance.yml` is the other half: it checks
`/var/run/reboot-required` on each node and, only where it's set, cordons and
drains the node, reboots it, waits for it to report `Ready` again, then
uncordons it — one node at a time (`serial: 1`), so patching the cluster never
means dropping every workload at once. Manual-only for now, same as
`playbooks/k3s.yml`:

```sh
ansible-playbook playbooks/k3s_maintenance.yml
```

Eventually this is meant to run on a schedule from a self-hosted Forgejo
Actions runner rather than by hand — not wired up yet, see the header comment
in the playbook.

#### Metrics (Prometheus)

`playbooks/k3s.yml` also deploys `roles/k3s_monitoring`: a lean
kube-prometheus-stack (Prometheus + prometheus-operator + node-exporter +
kube-state-metrics — no Grafana, no Alertmanager) so tools like OpenLens can
show node/pod metrics. It's installed as a `HelmChart` custom resource
dropped into k3s's own auto-deploying manifests directory
(`/var/lib/rancher/k3s/server/manifests/`) rather than run through a `helm`
binary — k3s's bundled helm-controller reconciles it the same way it
installs its own Traefik and ServiceLB, so this needs no extra Ansible
collection or kubeconfig on the controller. See
`roles/k3s_monitoring/defaults/main.yml` for the resource sizing (tuned for
a Raspberry Pi 4, not a real node) and why the control-plane component
scrapers (`kubeControllerManager`, `kubeScheduler`, `kubeProxy`, `kubeEtcd`)
are disabled — k3s doesn't expose those the way kube-prometheus-stack
expects, so leaving them on just produces permanently-"down" targets.

To point OpenLens at it: open the cluster's Settings → Metrics, set
Prometheus to "Auto detect" or explicitly to the `Operator` provider — it
should find the `kube-prometheus-stack-prometheus` service in the
`monitoring` namespace via the API server proxy, the same path OpenLens
already uses to reach the cluster, so nothing needs to be exposed outside
it.

#### LoadBalancer IPs (MetalLB)

`playbooks/k3s.yml` also deploys `roles/k3s_metallb`: MetalLB in L2 mode,
handing out real LAN IPs to `type: LoadBalancer` Services instead of the
`ClusterIP`-only world k3s would otherwise leave homelab-utils apps in. It's
installed the same way as monitoring — a `HelmChart` CR for k3s's
helm-controller to reconcile — plus a plain `IPAddressPool`/`L2Advertisement`
manifest dropped in the same directory; k3s's deploy controller applies both
kinds of file and retries the config manifest until the HelmChart's CRDs
exist.

MetalLB replaces k3s's bundled ServiceLB (Klipper), it doesn't run alongside
it — both would otherwise compete to satisfy the same Services. That's why
`inventory/group_vars/k3s_cluster.yml` sets `k3s_server_extra_args:
['--disable=servicelb']` — a separate, server-only var from `k3s_extra_args`,
since `k3s agent` doesn't understand `--disable` and would fail to start if
it were passed there too. The IP pool itself is
`k3s_metallb_address_range` in the same file — a range on the cluster's LAN
(`192.168.50.0/24`) that DHCP and every static assignment in
`inventory/hosts.yml` steer clear of. There's no built-in default: the role
fails fast if it's still empty.

Enabling `--disable=servicelb` on an already-running cluster doesn't need a
separate step: `roles/k3s_node` notices its exec line has changed and
reinstalls (restarting just the `k3s` service, not the node) the next time
`playbooks/k3s.yml` runs — see `CLAUDE.md` → "Key decisions".

#### Postgres (CloudNativePG)

`playbooks/k3s.yml` also deploys `roles/k3s_postgres`: CloudNativePG (CNPG),
a Postgres operator, installed the same way as monitoring and MetalLB — a
`HelmChart` CR for k3s's helm-controller to reconcile — plus a plain
`Cluster` CR (the actual database) and a `Secret` (superuser credentials),
dropped in the same auto-deploying manifests directory. This is a third
platform for `src/shared/postgres/`, alongside the Unraid Compose stack and
the Proxmox placeholder: `roles/k3s_postgres` layers
`src/shared/postgres/common/vars.yml` with
`src/shared/postgres/ansible/kubernetes/vars.yml` the same way
`compose_stack`/`lxc_app` layer an app's config, and pulls the superuser
password from the same `homelab/shared/postgres` Vault path those platforms
already use (see `docs/vault-secrets.md`) — a separate physical instance,
not a shared login across platforms, just the same identity.

1 primary + 1 replica, kept off the control-plane Pi via `nodeAffinity` (see
`roles/k3s_postgres/templates/postgres-cluster.yaml.j2`) — CNPG's own pod
anti-affinity then spreads the two across the 3 worker Pis. Storage is k3s's
default `local-path` StorageClass; that's node-local, but resilience here
comes from CNPG's own streaming replication between instances, not from
shared storage, so losing one instance's disk doesn't lose the data as long
as the other instance is up. HA replication only for now — no
`ScheduledBackup` to the NAS or anywhere else yet.

To connect: `kubectl -n shared-postgres get svc` lists four Services. Three
are CNPG's own, all `ClusterIP` — `shared-postgres-rw` (the current primary),
`-ro` (replicas only) and `-r` (any instance). An app running on the cluster
uses the first of those by DNS and needs nothing else:

```
shared-postgres-rw.shared-postgres.svc.cluster.local:5432
```

The fourth, `shared-postgres-lb`, is a `LoadBalancer` on
`k3s_postgres_loadbalancer_ip` (`inventory/group_vars/k3s_cluster.yml` —
inside the MetalLB pool above), which is how the cluster is reached from
outside it:

```sh
psql -h 192.168.50.81 -U postgres          # password: homelab/shared/postgres
```

It's declared in the `Cluster` CR as a CNPG *managed service* rather than as
a Service manifest of our own, so the operator keeps its selector pointed at
whichever instance is currently primary — a hand-written Service would need
re-pointing by hand after a failover. Credentials are the
`shared-postgres-superuser-vault` Secret Ansible renders; CNPG separately
auto-creates `shared-postgres-app` for the default `app` database it
bootstraps, which nothing uses yet.

The `-vault` suffix keeps that Secret clear of `<cluster>-superuser`, the
name CNPG uses for the superuser secret it generates itself. Name our own
object that and the operator treats it as one it already authored, so a
rotated password reaches the Secret and never reaches the database — the
Secret reads correctly, `psql` from the LAN keeps failing authentication, and
only `select rolpassword is null from pg_authid where rolname='postgres'`
inside the pod shows the disagreement.

That LoadBalancer is what makes the controller-reachable `host:port` the
`community.postgresql` tasks need — the same ones `compose_stack` and
`lxc_app` use to create an app's database and role — available here too, and
`roles/k3s_app` now uses exactly those tasks for any app declaring a `db:`
block.

Note the deliberate asymmetry, which reads like a mistake until you know why:
provisioning uses `192.168.50.81` (the LoadBalancer) because it runs on the
Ansible controller, which is off-cluster and can't route to a ClusterIP;
apps use `shared-postgres-rw.shared-postgres.svc.cluster.local` because
sending pod traffic out to the LAN and back would put MetalLB's L2 speaker in
the path of every query for nothing. Two addresses, one database, each
correct for its caller.

Without `kubectl` to hand, or if MetalLB is having a bad day, the in-cluster
routes still work:

```sh
kubectl -n shared-postgres port-forward svc/shared-postgres-rw 5432:5432
kubectl -n shared-postgres exec -it shared-postgres-1 -- psql -U postgres
```

**No backups.** HA replication only — a streaming replica on another Pi,
which covers a dead SD card and not a dropped table, a bad migration, or a
mistyped `DROP`. This mattered less when the cluster held nothing; it matters
now that Authentik's database lives here rather than on Unraid. A CNPG
`ScheduledBackup` to the NAS is the obvious next piece of work and isn't
built yet.

#### TLS (cert-manager)

`playbooks/k3s.yml` also applies `roles/k3s_cert_manager`: cert-manager plus
one `ClusterIssuer` named `letsencrypt`, both via the usual `HelmChart` CR +
plain manifest pair. An app's Ingress opts in with a single annotation —

```yaml
annotations:
  cert-manager.io/cluster-issuer: letsencrypt
```

— and cert-manager creates and renews the certificate on its own.

Two values have no default and the role refuses to run without them, both in
`inventory/group_vars/k3s_cluster.yml`:

- `k3s_cert_manager_acme_email` — an ACME account is registered against it.
- `k3s_cert_manager_solver` — the DNS-01 stanza for your DNS provider,
  rendered into the issuer as-is. A commented Cloudflare example is in that
  file; for anything else take the stanza from
  [cert-manager's docs](https://cert-manager.io/docs/configuration/acme/dns01/).
  Its API token goes in Vault at `homelab/k3s-cert-manager`.

DNS-01 rather than HTTP-01 because HTTP-01 needs Let's Encrypt to reach this
cluster from the internet on port 80, which it can't. That has a useful
consequence: a certificate can be issued **before** DNS points at the
cluster, so a migration's TLS is settled before its cutover rather than
after.

While working out a solver, point `k3s_cert_manager_acme_server` at Let's
Encrypt staging — production allows five failed validations per hostname per
hour, and exhausting it means waiting, not retrying. Check it registered:

```sh
kubectl get clusterissuer letsencrypt -o jsonpath='{.status.conditions[*].message}'
```

#### Dashboard (Traefik)

Traefik is the one chart here this repo doesn't install — k3s installs it
itself. `roles/k3s_traefik` only adjusts it, through a `HelmChartConfig`
merged over k3s's own `HelmChart`. Editing
`/var/lib/rancher/k3s/server/manifests/traefik.yaml` on the node instead
works until the next server restart rewrites it.

A stock k3s answers **404** on the dashboard, which looks like a broken
install and isn't: Traefik still builds the dashboard, but the Traefik chart
stopped creating the router that reaches it in v28. The role puts that router
back on Traefik's internal `traefik` entrypoint, which isn't published on the
Service — so it's reachable by port-forward and nothing else:

```sh
kubectl -n kube-system port-forward deploy/traefik 9000:9000
# then http://127.0.0.1:9000/dashboard/  — the trailing slash is required
```

Two things to check if that still 404s. `curl -i http://127.0.0.1:9000/ping`
returning 200 means the port-forward is fine and only the router is missing
(so the role hasn't run, or its `HelmChartConfig` hasn't reconciled yet —
`kubectl -n kube-system get ingressroute` should list `traefik-dashboard`).
And forward to `deploy/traefik`, not `svc/traefik`: the Service only publishes
80/443, so going through it lands you on the `web` entrypoint, where an
unmatched request also returns 404.

**Publishing it on a hostname** is one commented line in
`inventory/group_vars/k3s_cluster.yml`:

```yaml
k3s_traefik_dashboard_host: traefik.turtlesystems.uk
```

That renders an `IngressRoute` on `websecure` with a cert-manager certificate
and an Authentik forward-auth middleware in front. `k3s_traefik_dashboard_auth_address`
must be set too — the role refuses to run otherwise rather than publishing an
unauthenticated view of every router, service and middleware on the cluster.
It's already set in that file, so in practice this is the single line above
plus a DNS record pointing at Traefik's MetalLB address
(`kubectl -n kube-system get svc traefik`).

On the Authentik side, first: a Proxy Provider in **forward auth (domain
level)** mode, assigned to an application, added to the embedded outpost.
Domain level rather than single-application because the latter needs
`/outpost.goauthentik.io/` routed to Authentik on the dashboard's own
hostname, which from `kube-system` is a cross-namespace service reference
Traefik rejects by default.

An `IngressRoute` rather than an `Ingress`, unlike every app here, because
the dashboard is served by Traefik's internal `api@internal` and has no
Kubernetes Service for an `Ingress` to point at. That's also why its
certificate is an explicit `Certificate` resource — cert-manager watches
`Ingress`, not `IngressRoute`.

Clearing the hostname again is a real teardown: the role deletes the
manifest, and k3s's deploy controller garbage-collects the route, middleware
and certificate it created.

The port-forward stays enabled alongside the published route on purpose. Once
the dashboard is behind Authentik it depends on Authentik, which depends on
CNPG, which depends on a healthy cluster — the things you'd open the
dashboard to diagnose. Port-forward talks to the pod and traverses none of
it, so it's the break-glass path, not a leftover.

#### Apps on the cluster

Apps go in `k3s_apps:` in `inventory/group_vars/k3s_cluster.yml` — the k3s
equivalent of a host's `stacks:`/`apps:` list, in group_vars because an app
is deployed to the cluster rather than to a node. Same entry shape as the
other platforms (name, src, vault_path, optional `db:`, optional `state:`).

```sh
ansible-playbook playbooks/k3s.yml --tags apps
ansible-playbook playbooks/k3s.yml --tags apps -e only_apps=authentik
```

Ingress goes through the Traefik k3s already bundles, on the MetalLB address
its Service holds (`kubectl -n kube-system get svc traefik`) — so adding an
app costs a DNS record pointed at that one address, not a pool IP each.

Removal is `state: absent` on the entry, same tombstone rule as Unraid: keep
the entry, don't delete it, or nothing is torn down and the workload keeps
running unmanaged. Unlike Unraid it's a single pass — deleting the manifests
*is* the teardown, because k3s's deploy controller garbage-collects what each
file created — so `src/<app>/` can go in the same commit. The database, PVCs
and Vault path deliberately survive.

**Authentik** is currently the only app, and it got here by migrating off a
hand-made Unraid container rather than being deployed fresh. If you're
repeating that for something else, `docs/authentik-migration.md` is the
worked procedure; the part worth knowing up front is that the deployed
version must match the version a restored database was dumped from, because
Authentik runs its migrations on startup and they don't go backwards.

## SSH access

Ansible connects as `ansible_user: root` (set in
`build/config/ansible/inventory/hosts.yml`) over SSH. Password auth would mean either an interactive prompt every run —
which doesn't work from CI at all — or a plaintext password somewhere, so
use a dedicated key pair instead:

```sh
ssh-keygen -t ed25519 -f ~/.ssh/unraid_ansible -C "ansible@unraid-iac" -N ""
```

Install the public half as an authorized key on **every** Unraid host this
repo targets. Don't just `ssh-copy-id` it into `~/.ssh/authorized_keys` —
Unraid boots from the flash drive and `/root` lives on a RAM-backed overlay,
so anything written there is gone on the next reboot. Persist it through
`/boot/config/ssh/root.authorized_keys` instead, which Unraid copies into
place at every boot:

```sh
cat ~/.ssh/unraid_ansible.pub | ssh root@<unraid-host> \
  'cat >> /boot/config/ssh/root.authorized_keys'
```

For manual/bootstrap runs, load the key into `ssh-agent` and Ansible picks it
up automatically — no config file changes needed:

```sh
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/unraid_ansible
```

For CI, rather than adding yet another place secrets live, the private key
is stored in Vault (`homelab/ci/ssh`, see `docs/vault-secrets.md`) and the
workflow fetches it at the start of each run:

```sh
vault kv put kv/homelab/ci/ssh PRIVATE_KEY=@~/.ssh/unraid_ansible
```

**K3s Pis** work the same way but as a non-root `ansible` user, not root, and
with their own key pair (`~/.ssh/k3s_ansible`, Vault path
`homelab/ci/ssh-k3s`) — see `docs/vault-secrets.md` for the full commands.
Three differences from the Unraid steps above. The public key goes in the
ordinary `~/.ssh/authorized_keys` for the `ansible` user (Ubuntu's root
filesystem isn't rebuilt from flash on every boot, so there's no persistence
quirk to work around), and that user needs NOPASSWD sudo configured in
`/etc/sudoers.d/` — `playbooks/k3s.yml` installs k3s via `become: true`.

And the private half doesn't need loading into `ssh-agent` at all, even for a
manual run: `playbooks/k3s_ssh_key.yml` fetches it from Vault and writes it to
`local/k3s/homelab-utils.key`, and both `playbooks/k3s.yml` and
`playbooks/k3s_maintenance.yml` import that as their first play. So a k3s run
needs `VAULT_ADDR`/`VAULT_TOKEN` in the environment and nothing else — the
same "Vault is the only place secrets live" arrangement CI already uses for
the Unraid key, rather than a second setup step to remember. To fall back to a
local copy (Vault down, say), skip the fetch **and** point at the key —
overriding the path alone isn't enough, since the fetch play would still fail
before anything else ran:

```sh
ansible-playbook playbooks/k3s.yml \
  -e k3s_ssh_key_fetch=false \
  -e ansible_ssh_private_key_file=~/.ssh/k3s_ansible
```

## Bootstrapping (chicken-and-egg on the very first deploy)

CI/CD runs on Forgejo Actions — but Forgejo doesn't exist yet on the first
run, and Actions needs a self-hosted runner besides. So the first deploy is
manual:

1. Fill in `build/config/ansible/inventory/hosts.yml` with nas2's real
   address.
2. Set up the SSH key pair per "SSH access" above and install the public key
   on nas2 (and nas1, once you migrate anything to it).
3. `vault kv put` the secrets in `docs/vault-secrets.md` for
   `homelab/shared/postgres`, `homelab/forgejo`, and `homelab/ci/ssh`.
4. From WSL, with `VAULT_ADDR` / `VAULT_TOKEN` exported (`vault login` first)
   and the key loaded in `ssh-agent`:
   ```sh
   cd build/config/ansible
   ansible-playbook playbooks/unraid.yml --limit nas2
   ```
   This brings up shared Postgres, provisions the `forgejo` database/role,
   and brings up Forgejo. Use `unraid.yml` rather than `deploy.yml` at this
   point — no Proxmox guests exist yet, so the Proxmox play has nothing to
   target.
5. Log into Forgejo, create this repo there, push it.
6. Register a self-hosted Forgejo Actions runner reachable to Vault and to
   both Unraid hosts (labelled `unraid-deploy` — see
   `.forgejo/workflows/deploy.yml`). Give it an AppRole (`VAULT_ROLE_ID` /
   `VAULT_SECRET_ID`) scoped to read the `kv/homelab/*` paths it needs
   (including `homelab/ci/ssh`), stored as Forgejo Actions secrets alongside
   `VAULT_ADDR`.
7. From then on, pushes to `main` touching `src/**` or `build/config/ansible/**`
   deploy automatically — and only the stack(s) whose `src/<stack>/` folder
   actually changed (see "Deploying one platform, one host, one stack"
   below); a change under `build/config/ansible/` still triggers a full
   deploy of every stack on every host, since that's a change to how *all* of
   them get deployed.

### Bringing Proxmox online later

The shared Postgres and its LXC are the first thing to stand up here, and the
ordering is fiddly enough to have its own runbook: **`docs/postgres-proxmox.md`**.
The short version, and the reason it isn't just "run Terraform":

Terraform stores its state in Postgres, so provisioning the Postgres LXC with
Terraform would be circular. It's broken by keeping state on the
CloudNativePG cluster on the k3s Pis instead — something this configuration
has no hand in building — which means `terraform apply` now depends on the
cluster being up. That's a cross-platform dependency the rest of the repo
avoids, and it's worth it because the alternative is a local-state-then-migrate
dance that has to be got right exactly once.

After that, `docs/postgres-proxmox.md` walks through the template download,
`terraform apply -target=module.postgres`, checking the ZFS replication job
actually exists, installing Postgres with
`ansible-playbook playbooks/proxmox.yml -e only_stacks=postgres`, and
`playbooks/pve_host.yml` for node backups. It also covers failing over between
the two nodes and repointing clients afterwards.

Forgejo comes after all of that, on **turtle-proxmox-02** — its database is
the shared Postgres the runbook above creates, so none of it works until that
one is finished. `docs/forgejo-proxmox.md` walks through the extra Vault key
the native install needs, `terraform apply -target=module.forgejo`,
`ansible-playbook playbooks/proxmox.yml -e only_stacks=forgejo`, and the parts
Ansible deliberately does not do — moving repository data, and the DNS/SSH-port
cutover.

## Deploying one platform, one host, one stack

`playbooks/deploy.yml` runs everything; `playbooks/unraid.yml`,
`playbooks/proxmox.yml` and `playbooks/pve_host.yml` do one slice each
(Unraid stacks, Proxmox guests, and the Proxmox node's backup config
respectively). Within any of them, `--limit <host>`
narrows by host and `-e only_stacks=<name>[,<name>...]` narrows by stack or
app (name = the `name:` in the host's `stacks:`/`apps:` list, e.g. `forgejo`
or `postgres` — not the `src:` path):

```sh
# everything, both platforms
ansible-playbook playbooks/deploy.yml

# everything declared for nas2
ansible-playbook playbooks/unraid.yml --limit nas2

# just forgejo on nas2
ansible-playbook playbooks/unraid.yml --limit nas2 -e only_stacks=forgejo

# forgejo and its database, skipping any other stack on nas2
ansible-playbook playbooks/unraid.yml --limit nas2 -e only_stacks=forgejo,postgres
```

Note `only_stacks` doesn't resolve dependencies for you — if you limit to
`forgejo` before `postgres` has ever been deployed, the DB provisioning step
will fail because there's nothing to connect to. CI doesn't hit this: it
computes `only_stacks` from which `src/**` paths actually changed in the
push, so an unrelated app's push never touches Postgres or Forgejo.

`deploy.yml` runs Unraid before Proxmox by convention rather than necessity —
the two platforms no longer depend on each other. Proxmox-side apps provision
their databases against the Proxmox shared Postgres (192.168.50.54), and
Terraform keeps its state on the CloudNativePG cluster on k3s; neither goes
through Unraid. The node's backup config goes last, since a vzdump job pinned
to specific VMIDs needs those guests to exist. Note that `only_stacks` has no
meaning for `pve_host.yml` — it configures a node, not an app, and ignores it.

`playbooks/k3s.yml` is separate from all of the above — it isn't included in
`deploy.yml` and doesn't take `only_stacks`, because a cluster service there
isn't a `stacks:` entry to filter down to. It has two selectors of its own
instead: `--limit` narrows *which nodes*, and `--tags` narrows *which piece*.

```sh
# the whole cluster: nodes, then the cluster services, then the kubeconfig
ansible-playbook playbooks/k3s.yml

# re-converge one worker, e.g. after reimaging it
ansible-playbook playbooks/k3s.yml --limit k3s-node-02

# just one cluster service, no node install across 4 Pis
ansible-playbook playbooks/k3s.yml --tags postgres

# every cluster service, still no node install
ansible-playbook playbooks/k3s.yml --tags services
```

The tags, one per play (see the playbook's header comment): `nodes` (both
node plays), `metallb`, `monitoring`, `postgres`, `cert-manager`, `traefik`,
`services` (all five of those together), `apps`, `upgrades`, `kubeconfig`.
`services` and `apps` don't imply each other, so "everything except
reinstalling k3s" is `--tags services,apps`. The SSH-key fetch play is tagged
`always` rather than named, so it survives every filter — it's what any of
the others connect with, not something you'd pick.

Deploying a single service this way skips the `kubeconfig` play too, so
`local/`'s copy isn't refreshed. That only matters on a cluster rebuild;
add `--tags postgres,kubeconfig` if you want both.

Worth knowing what a tagged run does and doesn't tell you: every service role
just templates manifests into `/var/lib/rancher/k3s/server/manifests/`
for k3s's own controllers to reconcile, so the playbook finishing means the
files landed, not that the workload is up. Watch that separately, e.g.
`kubectl -n shared-postgres get cluster,pods -w`.

`playbooks/k3s_maintenance.yml` is separate again — see "Keeping it patched"
above. It ignores `--limit` grouping in one sense worth knowing: `serial: 1`
still applies to whatever `--limit` narrows the run to, so limiting to two
nodes still patches them one at a time, not together.

## Adding a new app

1. `src/<app>/common/vars.yml` — `env_defaults:` for the config that doesn't
   depend on where it runs.
2. Build the platform side(s) you need:

   **Unraid** — `src/<app>/ansible/unraid/` containing
   `docker-compose.yml` (static, `${VAR}`-driven, joining `unraid_shared`
   with `external: true` if it needs the shared Postgres), `vars.yml` of
   overrides, and `.env.example` documenting every var with secrets blank.
   Optionally an `icon.png` and the `net.unraid.docker.*` labels, so the
   Docker page shows something other than a question mark.

   **Proxmox** — `src/<app>/ansible/proxmox/` containing `vars.yml`,
   `install.yml`, and templates for the app's config file and systemd unit;
   plus `src/<app>/terraform/` defining its LXC, called from
   `build/config/terraform/main.tf`.

3. Add its `homelab/<app>` path to Vault (`docs/vault-secrets.md`).
4. Declare it on the target host — `stacks:` in an Unraid host's
   `host_vars/<host>.yml`, or `apps:` in a Proxmox guest's. Same entry shape
   either way (name, src, vault_path, optional `db:` block).
5. Push — or run the relevant playbook manually before CI exists.
6. Leave the platform you didn't build as a README placeholder rather than
   deleting the folder.

## Removing an app (Unraid)

**Deleting the entry from `stacks:` does not remove anything.** Ansible keeps
no record of what it deployed last run, so an app that disappears from the
list is simply never visited again — its containers keep running on the host,
now unmanaged and invisible to the playbook. This is the one place the Unraid
side doesn't behave like the Terraform side, where deleting a module call does
destroy the LXC.

Removal is therefore an instruction, not an absence. Set `state: absent` on
the entry and leave it in place:

```yaml
stacks:
  - name: arr
    src: arr
    vault_path: homelab/arr
    state: absent
```

The next run of `playbooks/unraid.yml` runs `docker compose down`, then
deletes the project folder (and with it the rendered `.env`). Because a
`host_vars/` edit is a change under `build/config/ansible/`, CI treats it as a
full converge — so pushing that change is enough to action the teardown.

Keep the entry as a tombstone until you're sure. It's the only record that
the app was deliberately removed rather than never deployed, and flipping
`state:` back to `present` redeploys it.

### What survives, and how to remove the rest

Bind mounts are untouched, so `/mnt/user/appdata/<app>` — which for a
SQLite-backed app like Shelfarr *is* the application — survives a teardown and
makes it reversible. External networks (`caddy-net`, `unraid_shared`) are left
alone too; they belong to Unraid or to another stack. Both copies of the
stack's icon *are* deleted without needing to be asked — the project-folder
one goes with the folder, and the one under `/mnt/user/appdata/icons/` is
removed explicitly. Neither is state; they're copies of a file in the repo,
and a redeploy puts them back.

Three things are destructive enough to stay opt-in, per entry:

| Key | Effect |
|---|---|
| `remove_volumes: true` | Also delete the project's named volumes. Bind mounts are unaffected either way. |
| `remove_images: local` | Also delete its images (`local` or `all`, as `docker compose down --rmi`). |
| `remove_database: true` | Drop the Postgres database **and** its role. Only for stacks with a `db:` block. |

Nothing removes the app's `homelab/<app>` path from Vault — that has its own
lifecycle, and destroying it would make the teardown irreversible. Delete it
by hand with `vault kv metadata delete` once you're finished with the app.

### Order of operations

`docker compose down` reads the compose file to know what it's removing, so
the teardown has to run **before** you delete `src/<app>/` or the host's
project folder. Delete those first and there's nothing left to tell Docker
what belonged to the project — you're cleaning up by hand with `docker rm` and
`docker network rm` instead. So: set `state: absent`, run the playbook, then
delete files in a second pass.

There's no equivalent on the Proxmox side yet. `lxc_app` only ever installs,
and a symmetric teardown needs a per-app `uninstall.yml` convention that no
app implements — the role is still a skeleton. For now, removing a Proxmox app
means stopping and disabling its systemd unit by hand, or destroying the
container via Terraform.

## Adding a host

**Unraid:** add it under `unraid_servers` in
`build/config/ansible/inventory/hosts.yml`, then create
`build/config/ansible/inventory/host_vars/<name>.yml` with its own `stacks:`
list — each host only runs what it's assigned.

**Proxmox:** define the LXC as a module in `src/<app>/terraform/`, call it
from `build/config/terraform/main.tf`, and `terraform apply`. The guest
appears in inventory automatically via the API — no `hosts.yml` edit — but
it still needs a `host_vars/<guest>.yml` declaring its `apps:` list.

**K3s:** add it as a new host under `k3s_control_plane` or `k3s_workers` (per
`inventory/hosts.yml` → `k3s_cluster`) — no `terraform apply`, the Pi has to
physically exist and have SSH access set up first (see "K3s (Raspberry Pi)"
above). No `host_vars/` entry needed beyond that: `roles/k3s_node` reads
`k3s_node_role` off the group, not the host, so which group a new Pi joins is
the only thing that decides whether it becomes another agent or a second
control-plane node — and this repo's role doesn't support the latter (single
server, no HA etcd) without changes to `k3s_node/tasks/server.yml`.