homelab/docs/vault-secrets.md

# Vault secret layout

KV v2 mount: `kv` (matches `vault_kv_mount` in
`build/config/ansible/inventory/group_vars/all.yml`).
One path per stack, fetched whole via `community.hashi_vault.vault_kv2_get`.

Paths are prefixed `homelab/`, not `unraid/`: an app may run as a Compose
stack on Unraid or natively in a Proxmox LXC, and its secrets are the same
either way. If you populated the old `unraid/*` paths before this rename,
move them and delete the originals:

```sh
for p in shared/postgres forgejo ci/ssh; do
  vault kv get -format=json "kv/unraid/$p" \
    | jq -r '.data.data | to_entries | map("\(.key)=\(.value)") | .[]' \
    | xargs vault kv put "kv/homelab/$p"
  vault kv metadata delete "kv/unraid/$p"
done
```

## Prerequisites

- [Vault CLI](https://developer.hashicorp.com/vault/docs/install) installed
  on whatever machine you're running these commands from (your workstation
  during bootstrap — not the Unraid box, and not the same thing as the
  Ansible controller needing `hvac`, though in practice it's the same WSL
  shell for both).
- `VAULT_ADDR` set to your Vault instance:

  ```sh
  export VAULT_ADDR='https://vault.example.internal:8200'
  ```

  If Vault is using a self-signed cert, either trust it properly or, for a
  homelab-only instance, set `export VAULT_SKIP_VERIFY=true` — don't do this
  against anything reachable outside your LAN.

## Login

Log in interactively once per shell session (or whenever the token expires);
the resulting token is cached to `~/.vault-token` and picked up automatically
by later `vault` commands and by `VAULT_TOKEN` if you export it.

Token auth (simplest, fine for a single-operator homelab):

```sh
vault login
# prompts for the token
```

If you're using a different auth backend, log in with that method instead —
adjust to whatever your Vault instance actually has enabled:

```sh
vault login -method=userpass username=<your-username>
vault login -method=oidc                                 # opens a browser
vault login -method=ldap username=<your-username>
```

Confirm the login worked and check the token's TTL:

```sh
vault token lookup
```

## One-time setup: enable the KV v2 mount

Only needed once, the first time this Vault instance is used for this repo
(skip if `kv/` already exists — check with `vault secrets list`):

```sh
vault secrets enable -path=kv -version=2 kv
```

## `homelab/shared/postgres`

| Key | Purpose |
|---|---|
| `POSTGRES_SUPERUSER` | Superuser name (matches `vars.yml` default `postgres`, but Vault is authoritative since it's paired with the password below) |
| `POSTGRES_SUPERUSER_PASSWORD` | Superuser password |

```sh
vault kv put kv/homelab/shared/postgres \
  POSTGRES_SUPERUSER=postgres \
  POSTGRES_SUPERUSER_PASSWORD='<generate-a-strong-password>'
```

Also seeds the k3s cluster's shared Postgres (`roles/k3s_postgres`,
CloudNativePG) — a separate physical instance from the Unraid/Proxmox ones,
but the same superuser identity, same path. See README.md → "Postgres
(CloudNativePG)".

## `homelab/forgejo`

| Key | Purpose |
|---|---|
| `DB_PASSWORD` | Password for the `forgejo` Postgres role (Ansible creates the role with this password) |
| `SECRET_KEY` | Forgejo `security.SECRET_KEY` |
| `INTERNAL_TOKEN` | Forgejo `security.INTERNAL_TOKEN` |
| `JWT_SECRET` | Forgejo `oauth2.JWT_SECRET` |
| `LFS_JWT_SECRET` | Forgejo `server.LFS_JWT_SECRET` — **Proxmox only**, see below |
| `OIDC_CLIENT_SECRET` | Authentik OAuth2 provider secret — **optional**, see below |

Generate the Forgejo secrets once and store them, rather than letting Forgejo
auto-generate on first boot — that keeps a from-scratch redeploy (fresh data
volume) reproducible instead of silently rotating tokens:

```sh
docker run --rm codeberg.org/forgejo/forgejo:10 forgejo generate secret
```

Run that four times (once per secret) and write them all together with the
DB password:

```sh
vault kv put kv/homelab/forgejo \
  DB_PASSWORD='<generate-a-strong-password>' \
  SECRET_KEY='<output-of-generate-secret>' \
  INTERNAL_TOKEN='<output-of-generate-secret>' \
  JWT_SECRET='<output-of-generate-secret>' \
  LFS_JWT_SECRET='<output-of-generate-secret>'
```

`LFS_JWT_SECRET` is only consumed by the Proxmox native install, which renders
`app.ini` in full and sets `INSTALL_LOCK` — nothing is left for Forgejo to
generate on first boot, and a value it invented for itself would be overwritten
on the next deploy anyway. The Unraid Compose stack doesn't reference it; the
key being present just means one extra unused line in the rendered `.env`. If
you only run the Unraid stack today, add it before the Proxmox cutover —
`src/forgejo/ansible/proxmox/install.yml` asserts on it rather than failing
three tasks later with an undefined-variable error:

```sh
vault kv patch kv/homelab/forgejo LFS_JWT_SECRET='<output-of-generate-secret>'
```

### `OIDC_CLIENT_SECRET`

Only needed if Forgejo signs users in through Authentik (step 8 of
`docs/forgejo-proxmox.md`). Unlike every other key on this path it is **not
read by Ansible** — nothing templates it, and `install.yml` does not assert on
it, so its absence is not an error. Forgejo keeps authentication sources in
its database, not in `app.ini`, so the value is consumed once by a
`forgejo admin auth add-oauth` run and lives in the database from then on.

It is kept here anyway, rather than only in Authentik, for the same reason as
the rest: a rebuild that restores from a `pg_dump` gets the auth source back
with the dump, but a rebuild from *nothing* has to re-run that command, and
this is where it looks for the value. Generated by Authentik when you create
the provider, not by you:

```sh
vault kv patch kv/homelab/forgejo OIDC_CLIENT_SECRET='<from-the-authentik-provider>'
```

The matching client ID is not a secret and is not stored here — it is an
argument to that one command, recorded in the runbook step alongside it.

## `homelab/arr`

| Key | Purpose |
|---|---|
| `SECRET_KEY_BASE` | Signs sessions and cookies |
| `ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY` | Encrypts every credential Shelfarr stores — indexer API keys, download client passwords, OIDC secrets |
| `ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY` | As above, for values Shelfarr needs to query on |
| `ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT` | As above |
| `SHELFARR_SETTING_OIDC_CLIENT_SECRET` | Client secret of the Authentik provider for Shelfarr |
| `PROWLARR_API_KEY` | Prowlarr's API key, for anything integrating with it |

One path for the whole stack, not one per container — `compose_stack` fetches
`vault_path` once and renders every key into a single `.env` that both
services read, so Prowlarr's key belongs here rather than under a path of its
own.

Shelfarr's entrypoint generates the first four on first run if they're unset, into
`/rails/storage/.secret_key_base` and `/rails/storage/.encryption_keys` — i.e.
into the appdata volume. Set them explicitly for the same reason Forgejo's
three secrets are fixed above: a from-scratch redeploy on a fresh volume would
otherwise generate new ones, and everything encrypted with the old values
becomes unreadable.

```sh
vault kv put kv/homelab/arr \
  SECRET_KEY_BASE="$(openssl rand -hex 64)" \
  ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY="$(openssl rand -base64 32)" \
  ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY="$(openssl rand -base64 32)" \
  ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT="$(openssl rand -base64 32)"
```

Set them **before the first start**, and set the three encryption keys as a
group — the entrypoint only falls back to its generated file when the primary
key is absent, so a partial set silently mixes provided and generated values.
Changing them after Shelfarr has stored anything means re-entering every
credential through Admin → Settings.

> **Do not add `RAILS_MASTER_KEY` here.** Despite the name it is not a
> generatable secret: it decrypts the `config/credentials.yml.enc` compiled
> into the upstream image, so only upstream's own key works. Supplying a value
> of your own aborts startup with
> `ActiveSupport::MessageEncryptor::InvalidMessage` during `db:prepare`.

The OIDC client secret is the one key here that isn't generated — copy it off
the Authentik provider:

```sh
vault kv patch kv/homelab/arr \
  SHELFARR_SETTING_OIDC_CLIENT_SECRET='<authentik-client-secret>'
```

`SHELFARR_SETTING_` is how upstream maps environment variables onto the
Admin → Settings store; the non-secret half of the OIDC config lives in
`src/arr/common/vars.yml`. Both halves have to be named in
`docker-compose.yml`'s `environment:` — `compose_stack` renders every Vault
key and every `env_defaults` entry into `.env`, but Compose reads `.env` only
to interpolate `${...}` in the compose file. A var that nothing references
never reaches the container, and Shelfarr just carries on with OIDC off.

Prowlarr's API key is generated the same way as Shelfarr's secrets, and for
the same reason — left unset, Prowlarr writes one of its own into
`/config/config.xml` on first start, where the only way to find it is the UI:

```sh
vault kv patch kv/homelab/arr PROWLARR_API_KEY="$(openssl rand -hex 16)"
```

Prowlarr binds config-file settings to environment as `PROWLARR__` plus the
section and key, so this reaches it as `PROWLARR__AUTH__APIKEY`. Releases
before the auth section was split read the same value as `PROWLARR__APIKEY`;
an unrecognised variable is ignored rather than fatal, so the symptom of the
wrong form is a key in the UI that doesn't match Vault, not a container that
won't start.

## `homelab/k3s-homelab-utils`

| Key | Purpose |
|---|---|
| `K3S_TOKEN` | Cluster join token — shared by the server and every agent |

Not app config — this is what makes `playbooks/k3s.yml` reproducible.
`k3s server` normally invents a token on first install and writes it to
`/var/lib/rancher/k3s/server/node-token`; every agent then has to be told that
value. Pinning it in Vault instead means a full rebuild (wipe both SD cards,
reinstall) reproduces the same cluster identity, because both the control
plane and every worker fetch the *same* value independently rather than one
generating it and handing it to the other.

Generate it once, before the first run of `playbooks/k3s.yml`:

```sh
vault kv put kv/homelab/k3s-homelab-utils \
  K3S_TOKEN="$(openssl rand -hex 32)"
```

Changing this after the cluster is up means re-running the playbook against
every node — the server re-issues its cert bundle around the new token and
every agent needs to reconnect with it, which is disruptive enough that
there's no automatic rotation path here, only a manual one.

## `homelab/authentik`

| Key | Purpose |
|---|---|
| `AUTHENTIK_SECRET_KEY` | Signs session cookies |
| `AUTHENTIK_POSTGRESQL__PASSWORD` | Password for the `authentik` role on the cluster's shared Postgres |

The first app on the k3s cluster, deployed by `roles/k3s_app` from the
`k3s_apps:` list in `inventory/group_vars/k3s_cluster.yml`. Both keys are
named for the environment variables Authentik itself reads, because
`roles/k3s_app` passes Vault keys through to the app's Kubernetes Secret
verbatim — so one value at `AUTHENTIK_POSTGRESQL__PASSWORD` serves both the
database provisioning step and the running app, with nothing restated.

```sh
vault kv put kv/homelab/authentik \
  AUTHENTIK_SECRET_KEY='<carried over from the Unraid deployment>' \
  AUTHENTIK_POSTGRESQL__PASSWORD="$(openssl rand -base64 32)"
```

`AUTHENTIK_SECRET_KEY` is the exception to "generate a fresh secret" — it
must be copied from the existing Unraid deployment, not invented. It signs
session cookies, so a new value logs every user out at the moment DNS moves.
See `docs/authentik-migration.md`, which is the only reason this path has a
"copy the old value" step at all; a from-scratch install would generate both.

## `homelab/k3s-cert-manager`

| Key | Purpose |
|---|---|
| `CLOUDFLARE_API_TOKEN` | DNS provider API token for cert-manager's ACME DNS-01 solver |

Consumed by `roles/k3s_cert_manager`, which renders every key at this path
into a Secret in the `cert-manager` namespace — so the key name above is a
convention, not a requirement: store whatever your DNS provider needs and
reference that name from `k3s_cert_manager_solver` in
`inventory/group_vars/k3s_cluster.yml`.

```sh
vault kv put kv/homelab/k3s-cert-manager \
  CLOUDFLARE_API_TOKEN='<token with Zone:DNS:Edit on the zone>'
```

DNS-01 rather than HTTP-01 because HTTP-01 needs Let's Encrypt to reach the
cluster from the internet on port 80, and this LAN deliberately isn't
reachable. Scope the token to the single zone it manages — cert-manager only
ever needs to write `_acme-challenge` TXT records.

## `homelab/ci/ssh`

Not consumed by `compose_stack` — this is the SSH private key Ansible itself
connects to the Unraid hosts with. It lives in Vault rather than as a
Forgejo Actions secret so there's one place secrets come from, not two; see
README.md "SSH access" for generating the key pair and installing the public
half on each host.

| Key | Purpose |
|---|---|
| `PRIVATE_KEY` | OpenSSH private key `ansible_user` (root) authenticates with |

```sh
# `@path` reads the value from a file, which handles the embedded newlines
# in a private key correctly — don't try to inline it as a normal KEY=value.
# Use $HOME, not `~`: this is a plain command argument, not a shell
# assignment, so bash won't tilde-expand a `~` sitting after the `@` — it'd
# get passed through literally and vault would look for a file named `~`.
vault kv put kv/homelab/ci/ssh PRIVATE_KEY=@"$HOME/.ssh/unraid_ansible"
```

## `homelab/ci/ssh-k3s`

The SSH private key Ansible connects to the 4 k3s Pis with. A separate key
pair from `homelab/ci/ssh` above, deliberately — the two host groups don't
share a trust boundary, so a leaked key for one shouldn't also be a working
key for the other. Same shape as the Unraid key otherwise:

| Key | Purpose |
|---|---|
| `PRIVATE_KEY` | OpenSSH private key the `ansible` user (per `inventory/hosts.yml` → `k3s_cluster`) authenticates with |

```sh
ssh-keygen -t ed25519 -f ~/.ssh/k3s_ansible -C "ansible@homelab-utils" -N ""
```

Create the `ansible` user on each of the 4 Pis with NOPASSWD sudo and install
the public half as its `authorized_keys` — see README.md "SSH access". Then:

```sh
vault kv put kv/homelab/ci/ssh-k3s PRIVATE_KEY=@"$HOME/.ssh/k3s_ansible"
```

Unlike the Unraid key above, nothing needs loading into `ssh-agent` to use
this: `playbooks/k3s_ssh_key.yml` reads this path and writes the key to
`local/k3s/homelab-utils.key` on the controller, and both `playbooks/k3s.yml`
and `playbooks/k3s_maintenance.yml` import it as their first play. A manual
run needs only `VAULT_ADDR`/`VAULT_TOKEN` in the environment (the "Login"
section above — no AppRole involvement yet, both k3s playbooks being
manual-only for now). See README.md "SSH access" for the fallback flags if
Vault is unreachable.

## `homelab/ci/proxmox`

Proxmox VE API token, used by two things: the `community.proxmox` dynamic
inventory plugin (`build/config/ansible/inventory/proxmox.yml`), and the
Terraform provider that creates LXCs. A token rather than a password — same
reasoning as the SSH key, a password means an interactive prompt or a
plaintext secret on disk.

Create the user and its token on either node (ACLs are cluster-wide), then
grant it a role. This is more involved than it looks, and the failure modes
give errors that don't obviously point at permissions:

```sh
pveum user add ansible@pam
pveum user token add ansible@pam terraform --privsep 0

pveum role add TerraformProv -privs "Datastore.Allocate \
  Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit \
  Pool.Allocate SDN.Use Sys.Audit Sys.Console Sys.Modify \
  VM.Allocate VM.Audit VM.Clone VM.Console VM.Migrate VM.PowerMgmt \
  VM.Replicate \
  VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory \
  VM.Config.Network VM.Config.Options"

pveum acl modify / --users 'ansible@pam' --roles TerraformProv
```

Three things that otherwise cost an afternoon:

- **`--privsep 0` is load-bearing.** A token created with privilege separation
  on (the UI default) carries its *own* ACL and ignores what the user was
  granted, so `pveum acl modify --users` looks like it did nothing. Either
  turn it off as above, or grant the token as well with
  `--tokens 'ansible@pam!terraform'`. With a purpose-made user rather than
  `root@pam` there is nothing gained by maintaining both layers — least
  privilege is already enforced on the user.
- **Privilege names are version-sensitive.** `VM.Monitor` appears in most
  guides for this and is rejected outright by current PVE (it is a QEMU
  monitor privilege, meaningless for LXC). `SDN.Use` is required from PVE 8.2
  on, where attaching a container to `vmbr0` is gated on
  `/sdn/zones/localnetwork/vmbr0`. Check against your own node rather than
  against any list, this one included: `pvesh get /access/roles/Administrator`
  prints the complete valid set.
- **`VM.Replicate` is there for the shared Postgres**, whose module creates a
  `pvesr` job (`replication_target_node` — see
  `src/shared/postgres/terraform/`). A module without replication won't need
  it.

Verify the grant landed before reaching for Terraform — this answers the
token's effective permissions, not the user's:

```sh
pveum user permissions ansible@pam --token terraform
```

Every `terraform apply` failure here is an HTTP 403 naming the exact missing
privilege, so the recovery loop is `pveum role modify TerraformProv --privs
"<name>" --append` and re-run. Note `--append`: without it, `--privs` replaces
the whole list rather than adding to it.

Then:

| Key | Purpose |
|---|---|
| `PROXMOX_URL` | API endpoint, e.g. `https://turtle-proxmox-01.home.turtlesystems.co.uk:8006/` |
| `PROXMOX_USER` | Token owner, e.g. `ansible@pam` |
| `PROXMOX_TOKEN_ID` | Token ID (the part after `!`) |
| `PROXMOX_TOKEN_SECRET` | Token secret — shown once at creation |

```sh
vault kv put kv/homelab/ci/proxmox \
  PROXMOX_URL='https://turtle-proxmox-01.home.turtlesystems.co.uk:8006/' \
  PROXMOX_USER='ansible@pam' \
  PROXMOX_TOKEN_ID='terraform' \
  PROXMOX_TOKEN_SECRET='<shown-once-at-creation>'
```

Ansible's inventory plugin reads these from the environment, so export them
before running anything that touches `proxmox_guests`. Terraform wants the
token in one combined string instead:

```sh
export TF_VAR_proxmox_api_token="${PROXMOX_USER}!${PROXMOX_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
```

## `homelab/ci/terraform`

Connection string for the Postgres database Terraform keeps its state in
(`backend "pg"` — see `build/config/terraform/README.md`). Separate from the
superuser credentials above: Terraform gets its own role, scoped to its own
database.

| Key | Purpose |
|---|---|
| `PG_CONN_STR` | `postgres://terraform:<password>@192.168.50.81:5432/terraform_state` |

The host is the **CloudNativePG cluster on k3s**
(`k3s_postgres_loadbalancer_ip`), not either shared Postgres this repo
deploys — Terraform cannot keep its state in a database it provisions itself.
Create the role and database once, against that cluster, then store the
string:

```sh
vault kv put kv/homelab/ci/terraform \
  PG_CONN_STR='postgres://terraform:<generate-a-strong-password>@192.168.50.81:5432/terraform_state'
```

## Verifying what was written

```sh
vault kv get kv/homelab/shared/postgres
vault kv get kv/homelab/forgejo
```

`vault kv put` (used above) replaces the whole secret at that path — to add
or change a single key without touching the others, use `vault kv patch`
instead:

```sh
vault kv patch kv/homelab/forgejo DB_PASSWORD='<new-password>'
```

## Adding a new stack's secrets

```sh
vault kv put kv/homelab/<stack-name> KEY1=value1 KEY2=value2
```

Then reference the same path as `vault_path` in the stack's entry under
`build/config/ansible/inventory/host_vars/<host>.yml`.

## AppRole for CI (Forgejo Actions)

The bootstrap commands above use your own human token. The Forgejo Actions
runner (`.forgejo/workflows/deploy.yml`) authenticates as an AppRole instead,
scoped to read-only on the `kv/homelab/*` paths. One-time setup:

```sh
# 1. Policy: read-only on every stack's secrets
vault policy write unraid-deploy - <<'EOF'
path "kv/data/homelab/*" {
  capabilities = ["read"]
}
EOF

# 2. Enable AppRole auth (skip if already enabled — check `vault auth list`)
vault auth enable approle

# 3. Create the role, bound to that policy
vault write auth/approle/role/unraid-deploy \
  token_policies="unraid-deploy" \
  token_ttl=15m \
  token_max_ttl=1h

# 4. Fetch the role ID (stable, not secret on its own)
vault read auth/approle/role/unraid-deploy/role-id

# 5. Generate a secret ID (treat as a secret — shown once)
vault write -f auth/approle/role/unraid-deploy/secret-id
```

Store the `role_id` and `secret_id` from steps 4–5 as `VAULT_ROLE_ID` and
`VAULT_SECRET_ID` in the Forgejo repo's Actions secrets (alongside
`VAULT_ADDR`) — that's what `deploy.yml` and
`build/config/ansible/inventory/group_vars/all.yml`
(`vault_auth_method: approle`) expect.

These never live in a file in this repo, in either mode:

- **CI:** Forgejo repo → Settings → Actions → Secrets. `deploy.yml` reads
  them from there via `${{ secrets.VAULT_ROLE_ID }}` /
  `${{ secrets.VAULT_SECRET_ID }}` and exports them as job env vars.
- **Manual runs** (bootstrap, or any ad hoc `ansible-playbook` invocation):
  export them as shell env vars first — `group_vars/all.yml` picks them up
  via `lookup('env', ...)`:

  ```sh
  export VAULT_AUTH_METHOD=approle
  export VAULT_ROLE_ID='<from step 4>'
  export VAULT_SECRET_ID='<from step 5>'
  ```

  (Bootstrap can also just use your own human token instead — leave
  `VAULT_AUTH_METHOD` unset, it defaults to `token`, and `vault login` from
  the "Login" section above is enough.)

Secret IDs can be regenerated (step 5) and old ones revoked without touching
the role itself if one ever leaks:

```sh
vault write auth/approle/role/unraid-deploy/secret-id-accessor/destroy \
  secret_id_accessor='<accessor-from-step-5-output>'
```