homelab/docs/postgres-proxmox.md

# Bringing up the shared Postgres on Proxmox

A run-once bootstrap, in the order the dependencies actually demand. Each step
exists because the one after it cannot start without it; if you already have a
piece, skip it and check the assertion at the end of the section.

The end state: a Debian LXC on the `AppData` ZFS pool of `turtle-proxmox-01`,
running Postgres 17, replicated every five minutes to `turtle-proxmox-02`, and
answering on 192.168.50.54:5432 for every app that declares a `db:`.

## Why this order

Two chains have to be satisfied before `terraform apply` will run at all:

- **State.** Terraform stores state in Postgres. Storing it in the database
  this configuration provisions would be circular, so it goes on the
  CloudNativePG cluster on the k3s Pis instead — a cluster Terraform has no
  hand in building. That cluster therefore has to be up first.
- **Inventory.** `playbooks/proxmox.yml` targets `proxmox_guests`, a group
  that only exists because of the API-backed dynamic inventory. Ansible cannot
  install into a container Terraform has just made until it can see it.

## 1. The k3s cluster and its CNPG instance

Needed only for Terraform state. If the cluster is already up, confirm the
LoadBalancer answers and move on.

```sh
cd build/config/ansible
ansible-playbook playbooks/k3s.yml --tags services
```

Then, against `k3s_postgres_loadbalancer_ip` (192.168.50.81), create the state
database and its role. This is the only hand-run SQL in the whole flow —
everything else provisions itself:

```sql
CREATE ROLE terraform LOGIN PASSWORD '<generate-a-strong-password>';
CREATE DATABASE terraform_state OWNER terraform;
```

Store the resulting connection string at `homelab/ci/terraform` in Vault (see
`vault-secrets.md`), then check it:

```sh
psql "postgres://terraform:$PG_PASSWORD@192.168.50.81:5432/terraform_state" -c '\conninfo'
```

## 2. Secrets

`homelab/shared/postgres` is the same path the Unraid and k3s instances
already use — one superuser identity for the "shared postgres" concept
wherever it runs. If it is populated, nothing to do:

```sh
vault kv get kv/homelab/shared/postgres
```

`homelab/ci/proxmox` needs a Proxmox API token, and `homelab/ci/ssh` the key
pair whose public half Terraform installs into the container.

## 3. The container template

On the node, once:

```sh
pveam available | grep debian-12
pveam download local debian-12-standard_12.7-1_amd64.tar.zst
```

## 4. Terraform

```sh
cd build/config/terraform
terraform init -backend-config="conn_str=$PG_CONN_STR"
```

Pin the provider to whatever this resolves and commit the resulting
`.terraform.lock.hcl`. Then:

```sh
export TF_VAR_proxmox_endpoint='https://turtle-proxmox-01.home.turtlesystems.co.uk:8006/'
export TF_VAR_proxmox_api_token='...'          # homelab/ci/proxmox
export TF_VAR_lxc_template_file_id='local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst'
export TF_VAR_ssh_public_keys='["'"$(cat ~/.ssh/unraid_ansible.pub)"'"]'

terraform apply -target=module.postgres
```

`-target` on purpose: the root module also defines the Forgejo LXC, and
bringing that up is a separate decision with its own migration story
(`inventory/host_vars/forgejo.yml`). Drop the flag once you want both.

If the apply 403s on `changing feature flags (except nesting) is only allowed
for root@pam`, the container already exists without nesting enabled and the
`ansible@pam` token cannot add it. Set it as root on the node, then re-plan —
it will read clean, and Postgres's systemd unit needs it:

```sh
ssh root@turtle-proxmox-01.home.turtlesystems.co.uk 'pct set 161 --features nesting=1'
ssh root@turtle-proxmox-01.home.turtlesystems.co.uk 'pct reboot 161'
```

Check the replication job exists — this is the step most easily missed,
because a container with no job looks identical in the storage view:

```sh
ssh root@turtle-proxmox-01 'pvesr status'
```

## 5. Install Postgres into it

```sh
cd build/config/ansible
ansible-inventory -i inventory/proxmox.yml --graph      # the guest should appear
ansible-playbook playbooks/proxmox.yml -e only_stacks=postgres
```

Then from the controller, confirming both that it listens on the LAN and that
the Vault password took:

```sh
psql "postgres://postgres:$PGPASSWORD@192.168.50.54:5432/postgres" -c 'SELECT version();'
```

## 6. Node backups

Replication covers a dead node, not a dropped table. `roles/pve_backup` writes
vzdump archives to the NAS:

```sh
ansible-playbook playbooks/pve_host.yml
```

## Growing the disk

Everything lives on the rootfs — that is what makes a replication snapshot
atomic (`src/shared/postgres/terraform/README.md`), so there is no second
volume to add when space runs short. Raise `disk_size` on the `postgres`
module in `build/config/terraform/main.tf` and apply.

Check the pool has the room first. Terraform will not: ZFS lets an apply
overcommit the pool happily, and the failure surfaces later as a write inside
the guest hitting ENOSPC.

```sh
ssh root@turtle-proxmox-01 'zpool list AppData; zfs list -o name,used,avail,refquota -r AppData'
ssh root@turtle-proxmox-02 'zpool list AppData'   # the target needs the room too
```

Then:

```sh
cd build/config/terraform
terraform plan     # expect an in-place update to disk.size, NOT a replacement
terraform apply
```

A **replacement** in that plan means something other than the size changed —
`prevent_destroy` will refuse it, which is the point. Never work around it
here; replacing this container destroys every database on it.

The resize itself is a `pct resize` of the rootfs, which on ZFS-backed storage
is a refquota change rather than a partition operation. It applies to a
running guest, ships no data, and needs nothing done inside the container
afterwards — no `resize2fs`, no Postgres restart. Confirm from inside:

```sh
ssh root@192.168.50.54 'df -h /'
```

Two things the larger number does *not* change: replication traffic, which is
a function of what actually gets written rather than of the quota, and vzdump
archive size, which covers used data only. Both track the databases, not the
headroom.

Shrinking is not available — it is a replacement, and `prevent_destroy` blocks
it — so overshoot rather than raising this every few months.

## Failing over

Replication makes the far copy a volume, not a running guest, so failover is
deliberate. Planned, with both nodes up, is an ordinary migration — fast,
because replication means only the delta has to ship:

```sh
ssh root@turtle-proxmox-01 'pct migrate 161 turtle-proxmox-02 --restart'
```

Unplanned, with the source node gone, means telling Proxmox to run the guest
from the replicated volume, and it loses every transaction committed since the
last successful send — up to `replication_schedule`, five minutes. Check what
you are about to accept first:

```sh
ssh root@turtle-proxmox-02 'pvesr status'      # look at "Last Sync"
```

Automating this is what a HA resource would add, and it is deliberately not
configured: automatic failover on a two-node cluster with no third vote is a
good way to get both nodes deciding they are the survivor.

## Afterwards: repointing clients

The address is written down in several files that cannot discover it. The
authoritative copy is `ip_address` on the `postgres` module in
`build/config/terraform/main.tf`, published as the `postgres_address` output.

| Where | What |
|---|---|
| `inventory/host_vars/<guest>.yml` | `db.provision_host` on each app that declares one |
| `src/<app>/ansible/proxmox/vars.yml` | that app's own `DB_HOST` |
| `src/<app>/ansible/unraid/vars.yml` | ditto, for anything still on Unraid — no `unraid_shared` network alias reaches this host |

Apps on the k3s cluster keep using the in-cluster CloudNativePG instance and
are unaffected.

## Migrating an existing database onto it

Per database, not `pg_dumpall` — the roles are recreated by the `db:` block on
each app's own entry, so only the data needs moving:

```sh
pg_dump -h <old-host> -U postgres -Fc forgejo > forgejo.dump
# create the role and database by deploying the app once with its `db:` block,
# then:
pg_restore -h 192.168.50.54 -U postgres -d forgejo --no-owner --role=forgejo forgejo.dump
```

`--no-owner --role=` rather than a straight restore: the dump carries
ownership from the old cluster, and the role provisioning in `lxc_app` /
`compose_stack` has already created the owner here with a Vault-managed
password. Restoring ownership from the dump would fight it.

Stop the application before dumping. A dump of a live database is consistent
as of its start, so anything written during it is silently absent from the
restore.