homelab/docs/vault-secrets.md

Vault secret layout

KV v2 mount: kv (matches vault_kv_mount in build/config/ansible/inventory/group_vars/all.yml). One path per stack, fetched whole via community.hashi_vault.vault_kv2_get.

Paths are prefixed homelab/, not unraid/: an app may run as a Compose stack on Unraid or natively in a Proxmox LXC, and its secrets are the same either way. If you populated the old unraid/* paths before this rename, move them and delete the originals:

for p in shared/postgres forgejo ci/ssh; do
  vault kv get -format=json "kv/unraid/$p" \
    | jq -r '.data.data | to_entries | map("\(.key)=\(.value)") | .[]' \
    | xargs vault kv put "kv/homelab/$p"
  vault kv metadata delete "kv/unraid/$p"
done

Prerequisites

• Vault CLI installed on whatever machine you're running these commands from (your workstation during bootstrap — not the Unraid box, and not the same thing as the Ansible controller needing hvac, though in practice it's the same WSL shell for both).

• VAULT_ADDR set to your Vault instance:

  export VAULT_ADDR='https://vault.example.internal:8200'
  

  If Vault is using a self-signed cert, either trust it properly or, for a homelab-only instance, set export VAULT_SKIP_VERIFY=true — don't do this against anything reachable outside your LAN.

Login

Log in interactively once per shell session (or whenever the token expires); the resulting token is cached to ~/.vault-token and picked up automatically by later vault commands and by VAULT_TOKEN if you export it.

Token auth (simplest, fine for a single-operator homelab):

vault login
# prompts for the token

If you're using a different auth backend, log in with that method instead — adjust to whatever your Vault instance actually has enabled:

vault login -method=userpass username=<your-username>
vault login -method=oidc                                 # opens a browser
vault login -method=ldap username=<your-username>

Confirm the login worked and check the token's TTL:

vault token lookup

One-time setup: enable the KV v2 mount

Only needed once, the first time this Vault instance is used for this repo (skip if kv/ already exists — check with vault secrets list):

vault secrets enable -path=kv -version=2 kv

homelab/shared/postgres

Key	Purpose
POSTGRES_SUPERUSER	Superuser name (matches vars.yml default postgres, but Vault is authoritative since it's paired with the password below)
POSTGRES_SUPERUSER_PASSWORD	Superuser password

vault kv put kv/homelab/shared/postgres \
  POSTGRES_SUPERUSER=postgres \
  POSTGRES_SUPERUSER_PASSWORD='<generate-a-strong-password>'

Also seeds the k3s cluster's shared Postgres (roles/k3s_postgres, CloudNativePG) — a separate physical instance from the Unraid/Proxmox ones, but the same superuser identity, same path. See README.md → "Postgres (CloudNativePG)".

homelab/forgejo

Key	Purpose
DB_PASSWORD	Password for the forgejo Postgres role (Ansible creates the role with this password)
SECRET_KEY	Forgejo security.SECRET_KEY
INTERNAL_TOKEN	Forgejo security.INTERNAL_TOKEN
JWT_SECRET	Forgejo oauth2.JWT_SECRET
LFS_JWT_SECRET	Forgejo server.LFS_JWT_SECRET — Proxmox only, see below
OIDC_CLIENT_SECRET	Authentik OAuth2 provider secret — optional, see below

Generate the Forgejo secrets once and store them, rather than letting Forgejo auto-generate on first boot — that keeps a from-scratch redeploy (fresh data volume) reproducible instead of silently rotating tokens:

docker run --rm codeberg.org/forgejo/forgejo:10 forgejo generate secret

Run that four times (once per secret) and write them all together with the DB password:

vault kv put kv/homelab/forgejo \
  DB_PASSWORD='<generate-a-strong-password>' \
  SECRET_KEY='<output-of-generate-secret>' \
  INTERNAL_TOKEN='<output-of-generate-secret>' \
  JWT_SECRET='<output-of-generate-secret>' \
  LFS_JWT_SECRET='<output-of-generate-secret>'

LFS_JWT_SECRET is only consumed by the Proxmox native install, which renders app.ini in full and sets INSTALL_LOCK — nothing is left for Forgejo to generate on first boot, and a value it invented for itself would be overwritten on the next deploy anyway. The Unraid Compose stack doesn't reference it; the key being present just means one extra unused line in the rendered .env. If you only run the Unraid stack today, add it before the Proxmox cutover — src/forgejo/ansible/proxmox/install.yml asserts on it rather than failing three tasks later with an undefined-variable error:

vault kv patch kv/homelab/forgejo LFS_JWT_SECRET='<output-of-generate-secret>'

OIDC_CLIENT_SECRET

Only needed if Forgejo signs users in through Authentik (step 8 of docs/forgejo-proxmox.md). Unlike every other key on this path it is not read by Ansible — nothing templates it, and install.yml does not assert on it, so its absence is not an error. Forgejo keeps authentication sources in its database, not in app.ini, so the value is consumed once by a forgejo admin auth add-oauth run and lives in the database from then on.

It is kept here anyway, rather than only in Authentik, for the same reason as the rest: a rebuild that restores from a pg_dump gets the auth source back with the dump, but a rebuild from nothing has to re-run that command, and this is where it looks for the value. Generated by Authentik when you create the provider, not by you:

vault kv patch kv/homelab/forgejo OIDC_CLIENT_SECRET='<from-the-authentik-provider>'

The matching client ID is not a secret and is not stored here — it is an argument to that one command, recorded in the runbook step alongside it.

homelab/arr

Key	Purpose
SECRET_KEY_BASE	Signs sessions and cookies
ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY	Encrypts every credential Shelfarr stores — indexer API keys, download client passwords, OIDC secrets
ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY	As above, for values Shelfarr needs to query on
ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT	As above
SHELFARR_SETTING_OIDC_CLIENT_SECRET	Client secret of the Authentik provider for Shelfarr
PROWLARR_API_KEY	Prowlarr's API key, for anything integrating with it

One path for the whole stack, not one per container — compose_stack fetches vault_path once and renders every key into a single .env that both services read, so Prowlarr's key belongs here rather than under a path of its own.

Shelfarr's entrypoint generates the first four on first run if they're unset, into /rails/storage/.secret_key_base and /rails/storage/.encryption_keys — i.e. into the appdata volume. Set them explicitly for the same reason Forgejo's three secrets are fixed above: a from-scratch redeploy on a fresh volume would otherwise generate new ones, and everything encrypted with the old values becomes unreadable.

vault kv put kv/homelab/arr \
  SECRET_KEY_BASE="$(openssl rand -hex 64)" \
  ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY="$(openssl rand -base64 32)" \
  ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY="$(openssl rand -base64 32)" \
  ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT="$(openssl rand -base64 32)"

Set them before the first start, and set the three encryption keys as a group — the entrypoint only falls back to its generated file when the primary key is absent, so a partial set silently mixes provided and generated values. Changing them after Shelfarr has stored anything means re-entering every credential through Admin → Settings.

Do not add RAILS_MASTER_KEY here. Despite the name it is not a generatable secret: it decrypts the config/credentials.yml.enc compiled into the upstream image, so only upstream's own key works. Supplying a value of your own aborts startup with ActiveSupport::MessageEncryptor::InvalidMessage during db:prepare.

The OIDC client secret is the one key here that isn't generated — copy it off the Authentik provider:

vault kv patch kv/homelab/arr \
  SHELFARR_SETTING_OIDC_CLIENT_SECRET='<authentik-client-secret>'

SHELFARR_SETTING_ is how upstream maps environment variables onto the Admin → Settings store; the non-secret half of the OIDC config lives in src/arr/common/vars.yml. Both halves have to be named in docker-compose.yml's environment: — compose_stack renders every Vault key and every env_defaults entry into .env, but Compose reads .env only to interpolate ${...} in the compose file. A var that nothing references never reaches the container, and Shelfarr just carries on with OIDC off.

Prowlarr's API key is generated the same way as Shelfarr's secrets, and for the same reason — left unset, Prowlarr writes one of its own into /config/config.xml on first start, where the only way to find it is the UI:

vault kv patch kv/homelab/arr PROWLARR_API_KEY="$(openssl rand -hex 16)"

Prowlarr binds config-file settings to environment as PROWLARR__ plus the section and key, so this reaches it as PROWLARR__AUTH__APIKEY. Releases before the auth section was split read the same value as PROWLARR__APIKEY; an unrecognised variable is ignored rather than fatal, so the symptom of the wrong form is a key in the UI that doesn't match Vault, not a container that won't start.

homelab/watchstate

No real secrets. WatchState creates its admin through a first-run web wizard, and each backend (Plex, Jellyfin) is added afterwards from the WebUI or docker exec -ti watchstate console, with its API token kept in /config — nothing is injected as an env var. But compose_stack fetches vault_path unconditionally, so the path still has to exist or the deploy fails at the lookup:

vault kv put kv/homelab/watchstate placeholder=unused

Same situation as homelab/jellyfin. Revisit if a future WatchState release grows a pre-settable API key or an OIDC client secret — add the key here and name it in src/watchstate/ansible/unraid/docker-compose.yml's environment: (a Vault key that nothing references never reaches the container).

homelab/k3s-homelab-utils

Key	Purpose
K3S_TOKEN	Cluster join token — shared by the server and every agent

Not app config — this is what makes playbooks/k3s.yml reproducible. k3s server normally invents a token on first install and writes it to /var/lib/rancher/k3s/server/node-token; every agent then has to be told that value. Pinning it in Vault instead means a full rebuild (wipe both SD cards, reinstall) reproduces the same cluster identity, because both the control plane and every worker fetch the same value independently rather than one generating it and handing it to the other.

Generate it once, before the first run of playbooks/k3s.yml:

vault kv put kv/homelab/k3s-homelab-utils \
  K3S_TOKEN="$(openssl rand -hex 32)"

Changing this after the cluster is up means re-running the playbook against every node — the server re-issues its cert bundle around the new token and every agent needs to reconnect with it, which is disruptive enough that there's no automatic rotation path here, only a manual one.

homelab/authentik

Key	Purpose
AUTHENTIK_SECRET_KEY	Signs session cookies
AUTHENTIK_POSTGRESQL__PASSWORD	Password for the authentik role on the cluster's shared Postgres

The first app on the k3s cluster, deployed by roles/k3s_app from the k3s_apps: list in inventory/group_vars/k3s_cluster.yml. Both keys are named for the environment variables Authentik itself reads, because roles/k3s_app passes Vault keys through to the app's Kubernetes Secret verbatim — so one value at AUTHENTIK_POSTGRESQL__PASSWORD serves both the database provisioning step and the running app, with nothing restated.

vault kv put kv/homelab/authentik \
  AUTHENTIK_SECRET_KEY='<carried over from the Unraid deployment>' \
  AUTHENTIK_POSTGRESQL__PASSWORD="$(openssl rand -base64 32)"

AUTHENTIK_SECRET_KEY is the exception to "generate a fresh secret" — it must be copied from the existing Unraid deployment, not invented. It signs session cookies, so a new value logs every user out at the moment DNS moves. See docs/authentik-migration.md, which is the only reason this path has a "copy the old value" step at all; a from-scratch install would generate both.

homelab/k3s-cert-manager

Key	Purpose
CLOUDFLARE_API_TOKEN	DNS provider API token for cert-manager's ACME DNS-01 solver

Consumed by roles/k3s_cert_manager, which renders every key at this path into a Secret in the cert-manager namespace — so the key name above is a convention, not a requirement: store whatever your DNS provider needs and reference that name from k3s_cert_manager_solver in inventory/group_vars/k3s_cluster.yml.

vault kv put kv/homelab/k3s-cert-manager \
  CLOUDFLARE_API_TOKEN='<token with Zone:DNS:Edit on the zone>'

DNS-01 rather than HTTP-01 because HTTP-01 needs Let's Encrypt to reach the cluster from the internet on port 80, and this LAN deliberately isn't reachable. Scope the token to the single zone it manages — cert-manager only ever needs to write _acme-challenge TXT records.

homelab/ci/ssh

Not consumed by compose_stack — this is the SSH private key Ansible itself connects to the Unraid hosts with. It lives in Vault rather than as a Forgejo Actions secret so there's one place secrets come from, not two; see README.md "SSH access" for generating the key pair and installing the public half on each host.

Key	Purpose
PRIVATE_KEY	OpenSSH private key ansible_user (root) authenticates with

# `@path` reads the value from a file, which handles the embedded newlines
# in a private key correctly — don't try to inline it as a normal KEY=value.
# Use $HOME, not `~`: this is a plain command argument, not a shell
# assignment, so bash won't tilde-expand a `~` sitting after the `@` — it'd
# get passed through literally and vault would look for a file named `~`.
vault kv put kv/homelab/ci/ssh PRIVATE_KEY=@"$HOME/.ssh/unraid_ansible"

homelab/ci/ssh-k3s

The SSH private key Ansible connects to the 4 k3s Pis with. A separate key pair from homelab/ci/ssh above, deliberately — the two host groups don't share a trust boundary, so a leaked key for one shouldn't also be a working key for the other. Same shape as the Unraid key otherwise:

Key	Purpose
PRIVATE_KEY	OpenSSH private key the ansible user (per inventory/hosts.yml → k3s_cluster) authenticates with

ssh-keygen -t ed25519 -f ~/.ssh/k3s_ansible -C "ansible@homelab-utils" -N ""

Create the ansible user on each of the 4 Pis with NOPASSWD sudo and install the public half as its authorized_keys — see README.md "SSH access". Then:

vault kv put kv/homelab/ci/ssh-k3s PRIVATE_KEY=@"$HOME/.ssh/k3s_ansible"

Unlike the Unraid key above, nothing needs loading into ssh-agent to use this: playbooks/k3s_ssh_key.yml reads this path and writes the key to local/k3s/homelab-utils.key on the controller, and both playbooks/k3s.yml and playbooks/k3s_maintenance.yml import it as their first play. A manual run needs only VAULT_ADDR/VAULT_TOKEN in the environment (the "Login" section above — no AppRole involvement yet, both k3s playbooks being manual-only for now). See README.md "SSH access" for the fallback flags if Vault is unreachable.

homelab/ci/proxmox

Proxmox VE API token, used by two things: the community.proxmox dynamic inventory plugin (build/config/ansible/inventory/proxmox.yml), and the Terraform provider that creates LXCs. A token rather than a password — same reasoning as the SSH key, a password means an interactive prompt or a plaintext secret on disk.

Create the user and its token on either node (ACLs are cluster-wide), then grant it a role. This is more involved than it looks, and the failure modes give errors that don't obviously point at permissions:

pveum user add ansible@pam
pveum user token add ansible@pam terraform --privsep 0

pveum role add TerraformProv -privs "Datastore.Allocate \
  Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit \
  Pool.Allocate SDN.Use Sys.Audit Sys.Console Sys.Modify \
  VM.Allocate VM.Audit VM.Clone VM.Console VM.Migrate VM.PowerMgmt \
  VM.Replicate \
  VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory \
  VM.Config.Network VM.Config.Options"

pveum acl modify / --users 'ansible@pam' --roles TerraformProv

Three things that otherwise cost an afternoon:

• --privsep 0 is load-bearing. A token created with privilege separation on (the UI default) carries its own ACL and ignores what the user was granted, so pveum acl modify --users looks like it did nothing. Either turn it off as above, or grant the token as well with --tokens 'ansible@pam!terraform'. With a purpose-made user rather than root@pam there is nothing gained by maintaining both layers — least privilege is already enforced on the user.
• Privilege names are version-sensitive. VM.Monitor appears in most guides for this and is rejected outright by current PVE (it is a QEMU monitor privilege, meaningless for LXC). SDN.Use is required from PVE 8.2 on, where attaching a container to vmbr0 is gated on /sdn/zones/localnetwork/vmbr0. Check against your own node rather than against any list, this one included: pvesh get /access/roles/Administrator prints the complete valid set.
• VM.Replicate is there for the shared Postgres, whose module creates a pvesr job (replication_target_node — see src/shared/postgres/terraform/). A module without replication won't need it.

Verify the grant landed before reaching for Terraform — this answers the token's effective permissions, not the user's:

pveum user permissions ansible@pam --token terraform

Every terraform apply failure here is an HTTP 403 naming the exact missing privilege, so the recovery loop is pveum role modify TerraformProv --privs "<name>" --append and re-run. Note --append: without it, --privs replaces the whole list rather than adding to it.

Then:

Key	Purpose
PROXMOX_URL	API endpoint, e.g. https://turtle-proxmox-01.home.turtlesystems.co.uk:8006/
PROXMOX_USER	Token owner, e.g. ansible@pam
PROXMOX_TOKEN_ID	Token ID (the part after !)
PROXMOX_TOKEN_SECRET	Token secret — shown once at creation

vault kv put kv/homelab/ci/proxmox \
  PROXMOX_URL='https://turtle-proxmox-01.home.turtlesystems.co.uk:8006/' \
  PROXMOX_USER='ansible@pam' \
  PROXMOX_TOKEN_ID='terraform' \
  PROXMOX_TOKEN_SECRET='<shown-once-at-creation>'

Ansible's inventory plugin reads these from the environment, so export them before running anything that touches proxmox_guests. Terraform wants the token in one combined string instead:

export TF_VAR_proxmox_api_token="${PROXMOX_USER}!${PROXMOX_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"

homelab/ci/terraform

Connection string for the Postgres database Terraform keeps its state in (backend "pg" — see build/config/terraform/README.md). Separate from the superuser credentials above: Terraform gets its own role, scoped to its own database.

Key	Purpose
PG_CONN_STR	postgres://terraform:<password>@192.168.50.81:5432/terraform_state

The host is the CloudNativePG cluster on k3s (k3s_postgres_loadbalancer_ip), not either shared Postgres this repo deploys — Terraform cannot keep its state in a database it provisions itself. Create the role and database once, against that cluster, then store the string:

vault kv put kv/homelab/ci/terraform \
  PG_CONN_STR='postgres://terraform:<generate-a-strong-password>@192.168.50.81:5432/terraform_state'

Verifying what was written

vault kv get kv/homelab/shared/postgres
vault kv get kv/homelab/forgejo

vault kv put (used above) replaces the whole secret at that path — to add or change a single key without touching the others, use vault kv patch instead:

vault kv patch kv/homelab/forgejo DB_PASSWORD='<new-password>'

Adding a new stack's secrets

vault kv put kv/homelab/<stack-name> KEY1=value1 KEY2=value2

Then reference the same path as vault_path in the stack's entry under build/config/ansible/inventory/host_vars/<host>.yml.

AppRole for CI (Forgejo Actions)

The bootstrap commands above use your own human token. The Forgejo Actions runner (.forgejo/workflows/deploy.yml) authenticates as an AppRole instead, scoped to read-only on the kv/homelab/* paths. One-time setup:

# 1. Policy: read-only on every stack's secrets
vault policy write unraid-deploy - <<'EOF'
path "kv/data/homelab/*" {
  capabilities = ["read"]
}
EOF

# 2. Enable AppRole auth (skip if already enabled — check `vault auth list`)
vault auth enable approle

# 3. Create the role, bound to that policy
vault write auth/approle/role/unraid-deploy \
  token_policies="unraid-deploy" \
  token_ttl=15m \
  token_max_ttl=1h

# 4. Fetch the role ID (stable, not secret on its own)
vault read auth/approle/role/unraid-deploy/role-id

# 5. Generate a secret ID (treat as a secret — shown once)
vault write -f auth/approle/role/unraid-deploy/secret-id

Store the role_id and secret_id from steps 4–5 as VAULT_ROLE_ID and VAULT_SECRET_ID in the Forgejo repo's Actions secrets (alongside VAULT_ADDR) — that's what deploy.yml and build/config/ansible/inventory/group_vars/all.yml (vault_auth_method: approle) expect.

These never live in a file in this repo, in either mode:

• CI: Forgejo repo → Settings → Actions → Secrets. deploy.yml reads them from there via ${{ secrets.VAULT_ROLE_ID }} / ${{ secrets.VAULT_SECRET_ID }} and exports them as job env vars.

• Manual runs (bootstrap, or any ad hoc ansible-playbook invocation): export them as shell env vars first — group_vars/all.yml picks them up via lookup('env', ...):

  export VAULT_AUTH_METHOD=approle
  export VAULT_ROLE_ID='<from step 4>'
  export VAULT_SECRET_ID='<from step 5>'
  

  (Bootstrap can also just use your own human token instead — leave VAULT_AUTH_METHOD unset, it defaults to token, and vault login from the "Login" section above is enough.)

Secret IDs can be regenerated (step 5) and old ones revoked without touching the role itself if one ever leaks:

vault write auth/approle/role/unraid-deploy/secret-id-accessor/destroy \
  secret_id_accessor='<accessor-from-step-5-output>'