Homelab IaC — Project Notes

Manages apps as infrastructure-as-code across three platforms — Unraid server(s), Proxmox, and a bare-metal k3s cluster — with secrets sourced from HashiCorp Vault and CI/CD via Forgejo Actions (self-hosted, on the Forgejo instance this repo deploys).

Ansible is the deployment tool on all three; what differs is the deployment primitive and who creates the host:

	Unraid	Proxmox	k3s
Provision the host	already exists	Terraform (`bpg/proxmox`) creates an LXC	already exists (4 Raspberry Pis)
Install the app	Ansible + `compose_stack` → `docker compose up -d`	Ansible + `lxc_app` → native install + systemd	Ansible + `k3s_app` → manifests into k3s's auto-deploy dir
App config source	`common/vars.yml` + `ansible/unraid/vars.yml`	`common/vars.yml` + `ansible/proxmox/vars.yml`	`common/vars.yml` + `ansible/kubernetes/vars.yml`
App list lives in	`host_vars/<host>.yml` `stacks:`	`host_vars/<guest>.yml` `apps:`	`group_vars/k3s_cluster.yml` `k3s_apps:`

The k3s column is the newest and reached that shape late: homelab-utils started as cluster-bootstrap-only, with no src/<app>/ entry, and the third column was added when Authentik moved onto it (see docs/authentik-migration.md). Two things about it don't reduce to the table:

The cluster itself has no src/ entry. Terraform never provisions it and it isn't an app, so it gets an inventory group (k3s_cluster) and roles/k3s_node, the same way the Proxmox node gets roles/pve_backup while its guests get lxc_app.
Cluster services aren't apps. k3s_metallb, k3s_monitoring, k3s_postgres and k3s_cert_manager are the cluster's own infrastructure — one hard-coded role each, config in group_vars. Apps go through the generic k3s_app and live in src/. Both end up as HelmChart CRs, so they look alike on disk; the distinction is whether anything would still need it if no app were deployed. See "K3s cluster (homelab-utils)" below.

Hosts

Two Unraid entries are gone from the original plan: nas2 was retired, and its address (192.168.50.2) was reassigned to turtle-proxmox-01 — so a stale copy of it fails by reaching the wrong machine rather than by timing out. What remains: nas01 (Unraid, 192.168.50.1), turtle-proxmox-01/-02 (Proxmox, .2/.3), the guests on them from the API inventory, and the four k3s Pis (.60–.63, MetalLB pool .80–.99). Gateway is 192.168.50.254.

Proxmox LXC guests live in 192.168.50.50–.59 — Forgejo .52, shared Postgres .54. Addresses here are assigned by hand from that band, not derived from anything, so a new guest's ip_address is a question for whoever owns the network rather than a value to pick.

The shared Postgres for everything outside the cluster is now the Proxmox LXC at 192.168.50.54 — see docs/postgres-proxmox.md. The Unraid Compose deployment under src/shared/postgres/ansible/unraid/ is kept for the platform-shape reason every unused platform folder is kept, not because it is running anywhere.

This repo supersedes the planning done in ../homelab/CLAUDE.md — that file has the original decision log if you want the "why" behind the architecture in more detail. Multi-host support, the shared-Postgres pattern, and the Forgejo-Actions-hosts-itself bootstrap problem were decided after that doc was written; this file and README.md are the current source of truth.

Layout

src/<app>/common/vars.yml — env_defaults: that hold regardless of where the app runs (version, ports, domain, DB name/user).
src/<app>/ansible/unraid/ — the Compose deployment: docker-compose.yml, .env.example, a vars.yml of Unraid-specific overrides (appdata paths, the shared-postgres Docker network alias), and optionally an icon.png for the Unraid Docker page.
src/<app>/ansible/proxmox/ — the native install: vars.yml of Proxmox-specific overrides, install.yml of app-specific install steps, and templates for the app's own config file and systemd unit. Two apps ship one: src/forgejo/ (binary + systemd unit, the shape the role was written for) and src/shared/postgres/ (distro packaging, so the unit comes from the package and install.yml only reconfigures the cluster).
src/<app>/ansible/kubernetes/ — the k3s deployment: vars.yml of cluster-specific overrides, and one or more *.yaml.j2 manifests (normally a HelmChart CR, plus anything the chart won't create — see src/authentik/). "kubernetes" rather than "k3s": the contents are plain Kubernetes apart from the HelmChart CR.
src/<app>/terraform/ — a module defining this app's LXC (template, cores, memory, disk, IP), called from build/config/terraform/main.tf.
src/shared/<service>/ — shared services (databases, etc.) used by multiple apps, same shape.
build/config/ansible/ — inventory, playbooks, and the roles. Three of them deploy apps, one per platform, and share the same layering and state: present|absent contract: compose_stack (Unraid), lxc_app (Proxmox guests), k3s_app (the cluster). compose_stack and k3s_app each split into deploy.yml/remove.yml, dispatched by main.yml on the entry's state:. The rest configure infrastructure rather than apps: k3s_node (bootstraps a Pi as k3s server or agent, picked by inventory group), k3s_metallb, k3s_monitoring, k3s_postgres, k3s_cert_manager (cluster services), k3s_maintenance (drain/reboot/ uncordon), unattended_upgrades, and pve_backup (the Proxmox node's backup storage and vzdump schedule).
build/config/terraform/ — Proxmox provider + backend "pg" state config, and the module calls saying what infrastructure exists.
.forgejo/workflows/ — CI/CD, runs once Forgejo + a self-hosted runner exist (see "Bootstrapping" in README.md — chicken-and-egg on the first deploy).

Key decisions

Compose files are static and reference ${VAR}; only the rendered .env changes per deploy. Nothing secret is ever committed.
Config is layered, not duplicated: common/vars.yml merged with ansible/<platform>/vars.yml (platform wins), then Vault secrets on top (secrets win). Both roles do this the same way, so moving an app between platforms means writing a new platform vars file, not restating its config.
One Vault KV path per app (homelab/<app>), fetched as a whole dict via community.hashi_vault.vault_kv2_get. The prefix is homelab/, not unraid/, because an app's secrets don't change with the platform it lands on.
Unraid draws a Compose deployment's stack row and its containers from unrelated sources, so one committed src/<app>/ansible/unraid/icon.png is copied to two places. The stack row is Compose Manager serving <project dir>/icon.png off disk — a file or nothing, no label and no URL, and only in the maintained fork (Compose Manager Plus); on the original plugin the copy is inert. The containers get icon, WebUI link and console shell from the dockerMan template that created them, which a Compose stack doesn't have, so each service carries net.unraid.docker.* labels (6.10+ falls back to them). The label path is a second copy under /mnt/user/appdata/icons/ rather than the project-folder one, because the webgui resolves it on every Docker page load and that shouldn't hit the flash drive.
Of those labels only the icon is role-supplied: compose_stack injects STACK_ICON underneath both vars.yml layers, so an app can override it with a hosted URL. WebUI and shell stay literal in the compose file — the WebUI port has to be the published one and the shell is a property of the base image, so both belong next to ports: and image: rather than in a variable. Details, including where SVG does and doesn't work, in README.md → "Icons on the Unraid Docker page".
Shared Postgres is provisioned once per host; per-app DB/user creation is idempotent via community.postgresql, not docker-entrypoint-initdb.d (which only runs once on an empty volume — wrong fit for a shared service apps get added to over time). The Unraid instance was pinned to 13 because its data directory predated this repo; the Proxmox one is 17 from the PGDG archive, a fresh cluster with nothing to stay compatible with.
The Proxmox shared Postgres lives on the AppData ZFS pool with a proxmox_replication job to the second node, and everything is on one volume — no mount_point splitting PGDATA from the WAL. Unlike Forgejo's "no bind mounts", which is about what vzdump includes, this is a correctness requirement: replication snapshots each of a guest's volumes independently, so two volumes replicate as two snapshots taken at different instants, and the far node can end up with a data directory and a WAL that disagree. One volume means one atomic snapshot — exactly the crash-consistent image WAL replay is designed for. It also means running out of space has one fix and not two: raise disk_size on the module call (256 GiB as of this writing, up from the original 32 once Forgejo's database made the OS-sized rootfs look optimistic), never add a second disk. Cheap to do — on ZFS the resize is a refquota change applied to a running guest — but one-way, since shrinking is a replacement and prevent_destroy blocks it. Runbook in docs/postgres-proxmox.md. Two further consequences worth keeping straight: replication is configured per guest, not per pool, so a container created on AppData without a job is a single point of failure that looks identical in the storage view (hence Terraform owning the job rather than a hand-run pvesr); and it's asynchronous, so a failover loses up to replication_schedule — five minutes — of commits. It is not a backup; pve_backup's vzdump is.
full_page_writes = off on the Proxmox instance, and only there. It guards against a torn page, which copy-on-write storage cannot produce, so on ZFS the full-page images are pure write amplification that also becomes replication traffic. The same setting on ext4 or xfs risks unrecoverable corruption — which is why it's in ansible/proxmox/vars.yml and must never migrate to common/vars.yml.
Terraform provisions the guest and stops there — it has no way to deploy an app into an LXC, so Ansible picks up from the point the container exists.
Proxmox guests keep all their state on their own rootfs; no bind mounts, no NFS shares mounted into a container. Counter-intuitive if the goal is "back the repos up to the NAS", but vzdump excludes bind mounts by design, so a bind-mounted repo directory is the one thing missing from the archive. Everything local means one archive is a complete filesystem restore point. Getting a copy off-box is the node's job (pve_backup → NFS storage on the NAS + a vzdump schedule), not the container's.
State an app keeps outside its container is that app's problem. Forgejo's database is on the shared Postgres LXC — a different guest, and so a different vzdump archive — so a systemd timer in the container pg_dumps it onto the container's own disk, and the vzdump archive carries both halves. The dump schedule and the vzdump window are a pair — move one, move the other.
Proxmox hosts come from the community.proxmox dynamic inventory plugin querying the Proxmox API, so terraform apply creating an LXC doesn't also require an inventory edit.
The terraform tag is what marks a Proxmox guest as this repo's to manage. The dynamic inventory returns every LXC on the cluster, hand-made ones included, so inventory/proxmox.yml filters on that tag — both Terraform modules set it (tags in their variables.tf), nothing else does. It filters rather than narrowing the proxmox_guests condition, so an unmanaged guest is absent from all as well and no future play can reach it by accident. The failure mode is silent in one direction: a new module that forgets the tag creates a container Ansible simply never visits, with no error anywhere — same shape as an omitted stacks: entry, and the first thing to check if a freshly-applied guest is never deployed to.
Terraform state lives in Postgres (backend "pg"), not a local file, so CI and a workstation see the same state — specifically the CloudNativePG cluster on k3s, not the shared Postgres this configuration provisions on Proxmox. That would be circular: Terraform would need the database to exist in order to create the container the database runs in. State goes somewhere Terraform has no hand in building, which breaks the cycle outright at the cost of a dependency the rest of the repo avoids — terraform apply now needs the Pis up. Worth it over the alternative (local state, apply, init -migrate-state), which has to be got right exactly once. Bootstrap ordering is docs/postgres-proxmox.md.
Every app keeps the same common/ + ansible/ + terraform/ shape even when only one platform is in use, so adding the other later doesn't require restructuring.
Removal on Unraid is explicit (state: absent on the stack entry), never implied by deleting the entry. Ansible stores no record of the last run, so an omitted stack isn't removed — it's just never visited again, and keeps running unmanaged. The entry stays as a tombstone documenting that the app was deliberately removed. Volumes, images and the app's database are further opt-ins (remove_volumes, remove_images, remove_database) rather than being implied by state: absent, and the Vault path is never touched — a default teardown is meant to be reversible. lxc_app has no counterpart yet; it would need a per-app uninstall.yml.
The k3s cluster's join token is a fixed value from Vault (homelab/k3s-homelab-utils → K3S_TOKEN), not the one k3s server generates for itself on first install. Both the control-plane play and the worker play look it up independently, so a full rebuild — wipe both SD cards, reinstall — reproduces the same cluster identity instead of needing a freshly-generated token fetched off the server and redistributed by hand. Same "generate once, store it, never let a redeploy invent a new one" reasoning as Forgejo's fixed secrets, applied to cluster identity instead of app state.
The SSH key the k3s plays connect with is fetched from Vault (homelab/ci/ssh-k3s) by playbooks/k3s_ssh_key.yml and written to local/, which both k3s.yml and k3s_maintenance.yml import as their first play — rather than the operator loading it into ssh-agent first, as the Unraid manual flow still expects. Same arrangement CI already uses for the Unraid key, just moved into the playbook because these two are manual-only and there's no workflow to hold the fetch step. It's a separate playbook, not a pre_tasks block, because both entry points need it and neither can assume the other ran. It targets k3s_cluster rather than localhost for a non-obvious reason: the implicit localhost isn't a member of all, so it wouldn't inherit vault_addr/vault_kv_mount/ vault_auth_method from group_vars/all.yml. Nothing in it connects to a Pi — gather_facts: false plus delegate_to: localhost, necessarily, since the key isn't on disk yet at that point. Falling back to a local key takes two overrides (-e k3s_ssh_key_fetch=false alongside -e ansible_ssh_private_key_file=...): pointing at another key alone doesn't help if Vault is what's broken, because the fetch play fails before any other play runs.
playbooks/k3s.yml is deliberately not wired into deploy.yml or .forgejo/workflows/deploy.yml. Converging 4 physical nodes on every push is a bigger blast radius than restarting a Compose stack — the same reasoning that keeps terraform apply manual for Proxmox. Run it by hand.
playbooks/k3s.yml's plays are tagged so one piece can be converged on its own — nodes, metallb, monitoring, postgres, services (those three cluster services together), upgrades, kubeconfig. Play-level tags, one per play, not per-task tags inside the roles: the plays already are the units, and tagging tasks would mean every role carrying tags for a selectivity only this playbook needs. This is the k3s equivalent of only_stacks for Unraid — a cluster service isn't a stacks: entry, so only_stacks can't reach it. The SSH-key import play is tagged always rather than getting a name of its own, since it's the prerequisite for anything else connecting and has to survive every --tags filter. Note what a tagged service run does not cover: the roles only drop manifests for k3s's controllers to reconcile, so a green playbook means the files landed, not that the workload came up.
k3s_node is one role, not two, even though a server and an agent install differently. Both need the same OS prep (cgroups, swap), and which install path runs is a group var (k3s_node_role: server/agent, set per inventory group) rather than a second role — so the shared prep can't drift between the two nodes types the way copy-pasted tasks eventually would.
Extra INSTALL_K3S_EXEC flags split across two vars, not one: k3s_extra_args (both roles) and k3s_server_extra_args (control-plane only). k3s agent doesn't understand server-only flags like --disable and fails to start if handed one, so a flag like --disable=servicelb (which k3s_metallb needs to avoid fighting k3s's bundled ServiceLB for the same IPs) can't safely live in the shared list.
k3s_node's install tasks track the exec line they last installed a node with (a small marker file next to k3s's own config) and reinstall — restarting the k3s/k3s-agent service, not the node — whenever it drifts from what k3s_extra_args/k3s_server_extra_args/k3s_api_tls_san currently say, even if k3s_version hasn't changed. Deliberately folded into the ordinary install task rather than a separate maintenance playbook: playbooks/k3s.yml is already manual-only and documented as safe to re-run, and unlike the reboots k3s_maintenance guards, bouncing the k3s process doesn't take pods down — containerd keeps them running underneath it. The worker play still runs with serial: 1 so at most one node's kubelet is ever mid-restart at once.
k3s server/k3s agent are installed with an explicit --node-name {{ inventory_hostname }}, not left to default to the OS hostname. This is what lets roles/k3s_maintenance address a node by inventory_hostname when draining/uncordoning — the k8s node object and the Ansible host are guaranteed to be the same string.
Patching a k3s Pi is split into two roles that don't know about each other directly, only through a file: unattended_upgrades installs updates hands-off but with Automatic-Reboot "false", and k3s_maintenance is the only thing that actually reboots a node — triggered by the presence of /var/run/reboot-required, which is the OS's own signal, not something either role invents. Splitting it this way means the risky half (taking a node out of the cluster) is exactly one thing (playbooks/k3s_maintenance.yml, serial: 1), not entangled with the routine half (installing packages), which runs unattended every day on every node without anyone thinking about it.
k3s_maintenance's kubectl calls (drain, wait, uncordon) are delegated to the control-plane host and run as k3s kubectl, k3s's own bundled client, rather than requiring a kubectl install or a kubeconfig anywhere else — including on whatever eventually runs this on a schedule. This also has to work when the node currently being processed is the control plane: delegating a host to itself is just a normal SSH connection, used right up until the moment that connection reboots out from under it.
k3s_monitoring (a lean kube-prometheus-stack, for OpenLens and similar tools) is installed as a HelmChart custom resource dropped into k3s's own auto-deploying manifests directory, not run through a helm binary or the kubernetes.core.helm collection — k3s ships a helm-controller that reconciles anything found there, the same mechanism it uses to install its own bundled Traefik and ServiceLB. That keeps the pattern this repo already uses everywhere else (Ansible renders a file, something else converges on it) instead of adding a second, unrelated way to reach the cluster from the controller. Grafana and Alertmanager are left disabled — OpenLens brings its own dashboards and this cluster doesn't page anyone — and the control-plane scrape targets (kubeControllerManager, kubeScheduler, kubeProxy, kubeEtcd) are disabled too, since k3s bundles those into one static binary instead of exposing them the way the chart expects; leaving them on just produces permanently-"down" targets, not a working scrape.
k3s_metallb (MetalLB, L2 mode) follows the same HelmChart CR pattern as k3s_monitoring, plus a second, plain manifest for its IPAddressPool/L2Advertisement config dropped in the same auto-deploying directory — k3s's deploy controller applies any manifest it finds there, not only HelmChart CRs, and retries one referencing CRDs that don't exist yet rather than failing outright, so the config doesn't need to wait on the chart install finishing first. It replaces k3s's bundled ServiceLB rather than running next to it — both would otherwise hand out IPs for the same LoadBalancer Services — so k3s_server_extra_args in inventory/group_vars/k3s_cluster.yml carries --disable=servicelb (server-only — k3s_extra_args, applied to both server and agent, can't carry a server-only flag like --disable without breaking agent installs). The IP pool (k3s_metallb_address_range, same file) has no built-in default; the role fails fast rather than silently advertising an empty pool.
k3s_postgres (CloudNativePG) is the k3s cluster's own shared Postgres — a third platform for src/shared/postgres/, alongside Unraid and Proxmox, deployed via Helm rather than Compose or a native install. Chosen over Bitnami's postgresql-ha (repmgr+pgpool, and Bitnami's free chart/image catalog was restructured into a "legacy" repo in 2025) and the Zalando operator (Patroni-based, heavier) because it fits the same HelmChart CR pattern already established: the operator installs as a chart CR, same shape as k3s_metallb/k3s_monitoring, and the actual cluster is a plain Cluster CR manifest — same "config manifest that outlives the CRDs it references" trick as k3s_metallb's IPAddressPool. It's a separate physical instance from the Docker-based shared/postgres (pinned to 13), but reuses the same Vault path (homelab/shared/postgres) rather than a k3s-specific one — one superuser identity for the "shared postgres" concept regardless of which platform it's running on, consistent with how every host already draws from that same path. 1 primary + 1 replica, scheduled on worker Pis only via nodeAffinity — the control plane stays free of app pods, same boundary k3s_metallb/k3s_monitoring already keep. HA replication only for now, no backups.
The CNPG cluster is published on the LAN through MetalLB at a pinned k3s_postgres_loadbalancer_ip, on top of the three ClusterIP Services CNPG makes for every Cluster. Declared inside the Cluster CR as a managed service (.spec.managed.services.additional, selectorType: rw) rather than as a Service manifest of this repo's own, so the operator owns the selector and the address follows a failover instead of needing to be re-pointed by hand. Pinned rather than auto-assigned because the address is meant to be written down in config elsewhere; it comes out of the bottom of k3s_metallb_address_range and would need a second autoAssign: false pool if anything else on this cluster ever wants a fixed IP. This is what closes the reachability half of per-app DB provisioning — the community.postgresql tasks compose_stack/lxc_app use can now reach this instance the same way they reach the Unraid one. roles/k3s_app is the other half, and uses exactly those tasks; note the asymmetry it relies on, which is easy to misread as a mistake. The provisioning runs from the Ansible controller and therefore uses the LoadBalancer address (k3s_postgres_loadbalancer_ip), because delegate_to: localhost is off the cluster and can't route to a ClusterIP. The app uses the in-cluster shared-postgres-rw DNS name, because sending pod traffic out to the LAN and back would put MetalLB's L2 speaker in the path of every query for no benefit. Two addresses for one database, each correct for its caller.
roles/k3s_app is the cluster's counterpart to compose_stack and lxc_app: the generic role that deploys an app, driven by k3s_apps: in group_vars/k3s_cluster.yml, as distinct from the k3s_* roles that each deploy one cluster service. It renders whatever *.yaml.j2 the app ships under src/<app>/ansible/kubernetes/ into k3s's auto-deploying manifests directory, so adding a manifest to an app is dropping a file next to the others rather than a role change. Secrets go into a separate Kubernetes Secret rendered from Vault, never interpolated into the manifests — the same committed-config/generated-secrets split as Unraid's static compose file plus rendered .env, which is why the manifests can stay 0644 on the node while the Secret is 0600.
k3s_apps: lives in group_vars/k3s_cluster.yml, not host_vars/, unlike both other platforms. An app is deployed to the cluster, not to a node: the play runs against k3s_control_plane only because that's where the manifests directory is, which is an implementation detail of how k3s is reached rather than a statement about where the app runs.
Removal on k3s is one pass, not two. compose_stack needs the compose file still on disk to run docker compose down, so state: absent and deleting src/<app>/ have to be separate commits (see "Removing an app" below). k3s's deploy controller tracks which resources each manifest file created, so deleting the file is the teardown and src/<app>/ can go in the same commit. The tombstone convention still applies for the same reason as everywhere else — an entry dropped from k3s_apps: is never visited again and keeps running unmanaged.
k3s_cert_manager is the fourth cluster service and the first that exists purely for apps rather than for the cluster — nothing in the cluster needs a certificate, k3s_app's tenants do. It's still a cluster service rather than an app, on the same reasoning as the shared Postgres: one ClusterIssuer that every app's Ingress annotates itself against, rather than each app carrying its own ACME account and DNS credentials. DNS-01, not HTTP-01, because HTTP-01 needs Let's Encrypt to reach the cluster from the internet on port 80 and this LAN deliberately isn't reachable — which has the useful side effect that a certificate can be issued before DNS points at the cluster, so a migration's TLS is settled before its cutover. The solver stanza itself has no default and the role fails fast without one: it depends on who runs the DNS, which the repo can't know. Same fail-fast-on-unset treatment as k3s_metallb_address_range.
Apps on k3s reach the outside through the Traefik k3s already bundles, on the MetalLB address its Service holds, rather than each getting its own LoadBalancer IP. One entrypoint and host-based routing means adding an app costs a DNS record, not a pool address — the opposite of the choice made for k3s_postgres, which is pinned to its own IP precisely because Postgres isn't HTTP and can't be name-routed.
k3s_traefik is the one cluster-service role that installs nothing: k3s installs Traefik itself, so the role only adjusts it, via a HelmChartConfig merged over k3s's own HelmChart rather than an edit to the traefik.yaml k3s rewrites on every server start. Its scope is the dashboard, which a stock k3s 404s — Traefik still builds it, but the chart stopped shipping the router that reaches it in v28. The role restores that router on Traefik's internal traefik entrypoint (port 9000, unpublished → port-forward only), and optionally publishes the dashboard on a hostname behind an Authentik forward-auth middleware when k3s_traefik_dashboard_host is set. Three things worth keeping straight: the internal route stays on even when the hostname one exists, because authenticated access depends on Authentik → CNPG → a healthy cluster, exactly what you'd open the dashboard to diagnose; the published route is an IngressRoute with an explicit cert-manager Certificate rather than an annotated Ingress, because the dashboard is api@internal and has no Service to point an Ingress at; and the role fails fast if a hostname is set without an auth address, since the failure mode is publishing every router, service and middleware on the cluster to the LAN unauthenticated. Authentik's provider must be forward auth (domain level) — single-application mode needs /outpost.goauthentik.io/ routed on the dashboard's own host, a cross-namespace service reference Traefik rejects unless allowCrossNamespace is on.

Adding a new app

src/<app>/common/vars.yml — env_defaults: for the portable config.
Whichever platform(s) it targets:
- Unraid: src/<app>/ansible/unraid/ with docker-compose.yml (static, ${VAR}-driven), vars.yml of overrides, .env.example documenting every var.
- Proxmox: src/<app>/ansible/proxmox/ with vars.yml, install.yml, and config/systemd templates; plus src/<app>/terraform/ defining its LXC, wired into build/config/terraform/main.tf.
- k3s: src/<app>/ansible/kubernetes/ with vars.yml and at least one *.yaml.j2 manifest — normally a HelmChart CR, plus anything the chart won't create for you. See src/authentik/ for a worked example, including the two things a chart usually leaves out: a PVC, and secrets referenced by secretKeyRef rather than set as values.
Populate its Vault path (homelab/<app>) with real secret values.
Declare it on the target:
- Unraid → host_vars/<host>.yml stacks: list.
- Proxmox → host_vars/<guest>.yml apps: list.
- k3s → group_vars/k3s_cluster.yml k3s_apps: list. All three take the same entry shape (name, src, vault_path, optional db:, optional state:). Never declare the same app on more than one at once — they share a Vault path and a database, so two live deployments corrupt each other's state. Moving an app between platforms is therefore a cutover with a verification step in the middle, not an edit; docs/authentik-migration.md is the worked example.
Leave the unused platforms' folders as README placeholders rather than deleting them — the shape is the point.

Removing an app

Unraid. Set state: absent on its stacks: entry and run playbooks/unraid.yml — don't delete the entry, which removes nothing. Then delete src/<app>/ in a second pass, never before: docker compose down needs the compose file to know what it's tearing down. Full detail, including the opt-in flags for volumes/images/database, is in README.md → "Removing an app".

k3s. Same state: absent tombstone rule, but one pass — deleting the manifests is the teardown, because k3s's deploy controller garbage-collects what each file created, so src/<app>/ can go in the same commit. The database, PVCs and Vault path deliberately survive; there's no remove_volumes/remove_database equivalent yet, and adding one is a "when it's actually wanted" job rather than a guess at the shape.

Proxmox. No teardown path at all — lxc_app would need a per-app uninstall.yml, and neither app that ships an install.yml has one. Note this is worse for the shared Postgres than for an ordinary app: tearing it down means every other app's database as well.

K3s cluster (homelab-utils)

4 Raspberry Pis, inventory/hosts.yml → k3s_cluster (k3s_control_plane: 1 host, k3s_workers: 3). No Terraform — the Pis already exist. The cluster itself has no src/<app>/ entry (it isn't an app), but apps now land on it: k3s_apps: in group_vars/k3s_cluster.yml, deployed by roles/k3s_app from src/<app>/ansible/kubernetes/. playbooks/k3s.yml is manual-only (see "Key decisions" above); run it with:

cd build/config/ansible
ansible-playbook playbooks/k3s.yml

# or one piece of it — see "Key decisions" above for the tag list
ansible-playbook playbooks/k3s.yml --tags postgres

# apps only, or one app
ansible-playbook playbooks/k3s.yml --tags apps
ansible-playbook playbooks/k3s.yml --tags apps -e only_apps=authentik

--tags services covers the cluster's own infrastructure and --tags apps what runs on top of it; neither implies the other, so "everything except reinstalling k3s" is --tags services,apps.

Prerequisites, once, before the first run: the ansible user exists on every Pi with NOPASSWD sudo and the public half of this repo's k3s SSH key installed, and homelab/k3s-homelab-utils → K3S_TOKEN plus homelab/ci/ssh-k3s → PRIVATE_KEY are set in Vault. Nothing needs setting up on the controller beyond VAULT_ADDR/VAULT_TOKEN — the private key comes from Vault at run time (see "Key decisions" above). Full detail in README.md → "K3s (Raspberry Pi)" and docs/vault-secrets.md.

Rebuilding: reimage the SD card(s), redo the ansible user + SSH key step, then re-run the playbook — the fixed Vault token means the rebuilt node(s) rejoin the same cluster identity rather than needing a token hunted down from a live server. Adding a fifth Pi: add it under k3s_control_plane or k3s_workers in hosts.yml; no host_vars/ entry needed, k3s_node_role comes from the group. This repo's k3s_node/server.yml assumes a single control-plane node — adding a second one for HA needs changes there, not just an inventory edit.

Patching: playbooks/k3s.yml also applies roles/unattended_upgrades to every node, so updates install themselves daily with no login required — but never reboot themselves (Automatic-Reboot "false"). Run playbooks/k3s_maintenance.yml to actually apply a pending reboot: it checks /var/run/reboot-required per node and, only where set, drains → reboots → waits for Ready → uncordons, serial: 1 so only one node is ever down. Manual-only for now, same as playbooks/k3s.yml — meant to eventually run on a schedule from a self-hosted Forgejo Actions runner rather than by hand.

Metrics: playbooks/k3s.yml also applies roles/k3s_monitoring, a lean kube-prometheus-stack (no Grafana/Alertmanager) so OpenLens and similar tools can show node/pod metrics — see "Key decisions" above for why it's a HelmChart CR rather than a helm install, and README.md → "Metrics (Prometheus)" for pointing OpenLens at it.

LoadBalancer IPs: playbooks/k3s.yml also applies roles/k3s_metallb, MetalLB in L2 mode, so type: LoadBalancer Services get real LAN IPs instead of staying ClusterIP-only. Set the pool it hands out from — k3s_metallb_address_range in inventory/group_vars/k3s_cluster.yml — to a range your DHCP scope and every static assignment in hosts.yml steer clear of before the first run; see "Key decisions" above for why it replaces ServiceLB rather than running alongside it, and README.md → "LoadBalancer IPs (MetalLB)" for more.

There's no node teardown path, unlike state: absent for apps — removing a Pi today means wiping its SD card and deleting its inventory entry by hand.

Shared Postgres: playbooks/k3s.yml also applies roles/k3s_postgres, CloudNativePG — the cluster's own shared Postgres, a third platform for src/shared/postgres/ alongside Unraid and Proxmox. See "Key decisions" above for why CloudNativePG and the topology, and README.md → "Postgres (CloudNativePG)" for connecting to it. Per-app database provisioning works now (roles/k3s_app, db: on a k3s_apps: entry) — note the two-address asymmetry described in "Key decisions". Still no backups: HA replication only, which protects against a dead SD card and not against a dropped table. That gap matters more now that Authentik's database lives here rather than inside whatever covers Unraid's /mnt/user/appdata; a CNPG ScheduledBackup is the obvious next piece of work.

TLS: playbooks/k3s.yml also applies roles/k3s_cert_manager — cert-manager plus one ClusterIssuer that every app's Ingress annotates itself against. Two values have no default and the role refuses to run without them: k3s_cert_manager_acme_email and k3s_cert_manager_solver (the DNS-01 stanza, which depends on your DNS provider — a commented Cloudflare example is in group_vars/k3s_cluster.yml). Use the Let's Encrypt staging directory while working a solver out; production allows five failed validations per hostname per hour.

Traefik dashboard: playbooks/k3s.yml also applies roles/k3s_traefik — the only cluster-service role that installs nothing, since k3s installs Traefik itself. Out of the box the dashboard is reachable only by port-forward (kubectl -n kube-system port-forward deploy/traefik 9000:9000, then http://127.0.0.1:9000/dashboard/ — trailing slash required; without this role a stock k3s 404s there). Publishing it on a hostname behind Authentik is one commented-out line, k3s_traefik_dashboard_host in group_vars/k3s_cluster.yml, plus a domain-level forward-auth Proxy Provider on the Authentik side. See "Key decisions" above and README.md → "Dashboard (Traefik)".

Apps: k3s_apps: in group_vars/k3s_cluster.yml, deployed by roles/k3s_app. Authentik is the first and currently only one — it moved off a hand-made Unraid container rather than being deployed fresh, so src/authentik/ansible/unraid/ is a README explaining why there's no compose file there. The cutover procedure is docs/authentik-migration.md; the part worth knowing without reading it is that the deployed AUTHENTIK_VERSION must match the version a restored database was dumped from, because Authentik's migrations run on startup and are one-way.

38 KiB Raw Blame History Unescape Escape