homelab/docs/postgres-proxmox.md

Bringing up the shared Postgres on Proxmox

A run-once bootstrap, in the order the dependencies actually demand. Each step exists because the one after it cannot start without it; if you already have a piece, skip it and check the assertion at the end of the section.

The end state: a Debian LXC on the AppData ZFS pool of turtle-proxmox-01, running Postgres 17, replicated every five minutes to turtle-proxmox-02, and answering on 192.168.50.54:5432 for every app that declares a db:.

Why this order

Two chains have to be satisfied before terraform apply will run at all:

• State. Terraform stores state in Postgres. Storing it in the database this configuration provisions would be circular, so it goes on the CloudNativePG cluster on the k3s Pis instead — a cluster Terraform has no hand in building. That cluster therefore has to be up first.
• Inventory. playbooks/proxmox.yml targets proxmox_guests, a group that only exists because of the API-backed dynamic inventory. Ansible cannot install into a container Terraform has just made until it can see it.

1. The k3s cluster and its CNPG instance

Needed only for Terraform state. If the cluster is already up, confirm the LoadBalancer answers and move on.

cd build/config/ansible
ansible-playbook playbooks/k3s.yml --tags services

Then, against k3s_postgres_loadbalancer_ip (192.168.50.81), create the state database and its role. This is the only hand-run SQL in the whole flow — everything else provisions itself:

CREATE ROLE terraform LOGIN PASSWORD '<generate-a-strong-password>';
CREATE DATABASE terraform_state OWNER terraform;

Store the resulting connection string at homelab/ci/terraform in Vault (see vault-secrets.md), then check it:

psql "postgres://terraform:$PG_PASSWORD@192.168.50.81:5432/terraform_state" -c '\conninfo'

2. Secrets

homelab/shared/postgres is the same path the Unraid and k3s instances already use — one superuser identity for the "shared postgres" concept wherever it runs. If it is populated, nothing to do:

vault kv get kv/homelab/shared/postgres

homelab/ci/proxmox needs a Proxmox API token, and homelab/ci/ssh the key pair whose public half Terraform installs into the container.

3. The container template

On the node, once:

pveam available | grep debian-12
pveam download local debian-12-standard_12.7-1_amd64.tar.zst

4. Terraform

cd build/config/terraform
terraform init -backend-config="conn_str=$PG_CONN_STR"

Pin the provider to whatever this resolves and commit the resulting .terraform.lock.hcl. Then:

export TF_VAR_proxmox_endpoint='https://turtle-proxmox-01.home.turtlesystems.co.uk:8006/'
export TF_VAR_proxmox_api_token='...'          # homelab/ci/proxmox
export TF_VAR_lxc_template_file_id='local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst'
export TF_VAR_ssh_public_keys='["'"$(cat ~/.ssh/unraid_ansible.pub)"'"]'

terraform apply -target=module.postgres

-target on purpose: the root module also defines the Forgejo LXC, and bringing that up is a separate decision with its own migration story (inventory/host_vars/forgejo.yml). Drop the flag once you want both.

If the apply 403s on changing feature flags (except nesting) is only allowed for root@pam, the container already exists without nesting enabled and the ansible@pam token cannot add it. Set it as root on the node, then re-plan — it will read clean, and Postgres's systemd unit needs it:

ssh root@turtle-proxmox-01.home.turtlesystems.co.uk 'pct set 161 --features nesting=1'
ssh root@turtle-proxmox-01.home.turtlesystems.co.uk 'pct reboot 161'

Check the replication job exists — this is the step most easily missed, because a container with no job looks identical in the storage view:

ssh root@turtle-proxmox-01 'pvesr status'

5. Install Postgres into it

cd build/config/ansible
ansible-inventory -i inventory/proxmox.yml --graph      # the guest should appear
ansible-playbook playbooks/proxmox.yml -e only_stacks=postgres

Then from the controller, confirming both that it listens on the LAN and that the Vault password took:

psql "postgres://postgres:$PGPASSWORD@192.168.50.54:5432/postgres" -c 'SELECT version();'

6. Node backups

Replication covers a dead node, not a dropped table. roles/pve_backup writes vzdump archives to the NAS:

ansible-playbook playbooks/pve_host.yml

Growing the disk

Everything lives on the rootfs — that is what makes a replication snapshot atomic (src/shared/postgres/terraform/README.md), so there is no second volume to add when space runs short. Raise disk_size on the postgres module in build/config/terraform/main.tf and apply.

Check the pool has the room first. Terraform will not: ZFS lets an apply overcommit the pool happily, and the failure surfaces later as a write inside the guest hitting ENOSPC.

ssh root@turtle-proxmox-01 'zpool list AppData; zfs list -o name,used,avail,refquota -r AppData'
ssh root@turtle-proxmox-02 'zpool list AppData'   # the target needs the room too

Then:

cd build/config/terraform
terraform plan     # expect an in-place update to disk.size, NOT a replacement
terraform apply

A replacement in that plan means something other than the size changed — prevent_destroy will refuse it, which is the point. Never work around it here; replacing this container destroys every database on it.

The resize itself is a pct resize of the rootfs, which on ZFS-backed storage is a refquota change rather than a partition operation. It applies to a running guest, ships no data, and needs nothing done inside the container afterwards — no resize2fs, no Postgres restart. Confirm from inside:

ssh root@192.168.50.54 'df -h /'

Two things the larger number does not change: replication traffic, which is a function of what actually gets written rather than of the quota, and vzdump archive size, which covers used data only. Both track the databases, not the headroom.

Shrinking is not available — it is a replacement, and prevent_destroy blocks it — so overshoot rather than raising this every few months.

Failing over

Replication makes the far copy a volume, not a running guest, so failover is deliberate. Planned, with both nodes up, is an ordinary migration — fast, because replication means only the delta has to ship:

ssh root@turtle-proxmox-01 'pct migrate 161 turtle-proxmox-02 --restart'

Unplanned, with the source node gone, means telling Proxmox to run the guest from the replicated volume, and it loses every transaction committed since the last successful send — up to replication_schedule, five minutes. Check what you are about to accept first:

ssh root@turtle-proxmox-02 'pvesr status'      # look at "Last Sync"

Automating this is what a HA resource would add, and it is deliberately not configured: automatic failover on a two-node cluster with no third vote is a good way to get both nodes deciding they are the survivor.

Afterwards: repointing clients

The address is written down in several files that cannot discover it. The authoritative copy is ip_address on the postgres module in build/config/terraform/main.tf, published as the postgres_address output.

Where	What
inventory/host_vars/<guest>.yml	db.provision_host on each app that declares one
src/<app>/ansible/proxmox/vars.yml	that app's own DB_HOST
src/<app>/ansible/unraid/vars.yml	ditto, for anything still on Unraid — no unraid_shared network alias reaches this host

Apps on the k3s cluster keep using the in-cluster CloudNativePG instance and are unaffected.

Migrating an existing database onto it

Per database, not pg_dumpall — the roles are recreated by the db: block on each app's own entry, so only the data needs moving:

pg_dump -h <old-host> -U postgres -Fc forgejo > forgejo.dump
# create the role and database by deploying the app once with its `db:` block,
# then:
pg_restore -h 192.168.50.54 -U postgres -d forgejo --no-owner --role=forgejo forgejo.dump

--no-owner --role= rather than a straight restore: the dump carries ownership from the old cluster, and the role provisioning in lxc_app / compose_stack has already created the owner here with a Vault-managed password. Restoring ownership from the dump would fight it.

Stop the application before dumping. A dump of a live database is consistent as of its start, so anything written during it is silently absent from the restore.