# Bringing up the shared Postgres on Proxmox A run-once bootstrap, in the order the dependencies actually demand. Each step exists because the one after it cannot start without it; if you already have a piece, skip it and check the assertion at the end of the section. The end state: a Debian LXC on the `AppData` ZFS pool of `turtle-proxmox-01`, running Postgres 17, replicated every five minutes to `turtle-proxmox-02`, and answering on 192.168.50.54:5432 for every app that declares a `db:`. ## Why this order Two chains have to be satisfied before `terraform apply` will run at all: - **State.** Terraform stores state in Postgres. Storing it in the database this configuration provisions would be circular, so it goes on the CloudNativePG cluster on the k3s Pis instead — a cluster Terraform has no hand in building. That cluster therefore has to be up first. - **Inventory.** `playbooks/proxmox.yml` targets `proxmox_guests`, a group that only exists because of the API-backed dynamic inventory. Ansible cannot install into a container Terraform has just made until it can see it. ## 1. The k3s cluster and its CNPG instance Needed only for Terraform state. If the cluster is already up, confirm the LoadBalancer answers and move on. ```sh cd build/config/ansible ansible-playbook playbooks/k3s.yml --tags services ``` Then, against `k3s_postgres_loadbalancer_ip` (192.168.50.81), create the state database and its role. This is the only hand-run SQL in the whole flow — everything else provisions itself: ```sql CREATE ROLE terraform LOGIN PASSWORD ''; CREATE DATABASE terraform_state OWNER terraform; ``` Store the resulting connection string at `homelab/ci/terraform` in Vault (see `vault-secrets.md`), then check it: ```sh psql "postgres://terraform:$PG_PASSWORD@192.168.50.81:5432/terraform_state" -c '\conninfo' ``` ## 2. Secrets `homelab/shared/postgres` is the same path the Unraid and k3s instances already use — one superuser identity for the "shared postgres" concept wherever it runs. If it is populated, nothing to do: ```sh vault kv get kv/homelab/shared/postgres ``` `homelab/ci/proxmox` needs a Proxmox API token, and `homelab/ci/ssh` the key pair whose public half Terraform installs into the container. ## 3. The container template On the node, once: ```sh pveam available | grep debian-12 pveam download local debian-12-standard_12.7-1_amd64.tar.zst ``` ## 4. Terraform ```sh cd build/config/terraform terraform init -backend-config="conn_str=$PG_CONN_STR" ``` Pin the provider to whatever this resolves and commit the resulting `.terraform.lock.hcl`. Then: ```sh export TF_VAR_proxmox_endpoint='https://turtle-proxmox-01.home.turtlesystems.co.uk:8006/' export TF_VAR_proxmox_api_token='...' # homelab/ci/proxmox export TF_VAR_lxc_template_file_id='local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst' export TF_VAR_ssh_public_keys='["'"$(cat ~/.ssh/unraid_ansible.pub)"'"]' terraform apply -target=module.postgres ``` `-target` on purpose: the root module also defines the Forgejo LXC, and bringing that up is a separate decision with its own migration story (`inventory/host_vars/forgejo.yml`). Drop the flag once you want both. If the apply 403s on `changing feature flags (except nesting) is only allowed for root@pam`, the container already exists without nesting enabled and the `ansible@pam` token cannot add it. Set it as root on the node, then re-plan — it will read clean, and Postgres's systemd unit needs it: ```sh ssh root@turtle-proxmox-01.home.turtlesystems.co.uk 'pct set 161 --features nesting=1' ssh root@turtle-proxmox-01.home.turtlesystems.co.uk 'pct reboot 161' ``` Check the replication job exists — this is the step most easily missed, because a container with no job looks identical in the storage view: ```sh ssh root@turtle-proxmox-01 'pvesr status' ``` ## 5. Install Postgres into it ```sh cd build/config/ansible ansible-inventory -i inventory/proxmox.yml --graph # the guest should appear ansible-playbook playbooks/proxmox.yml -e only_stacks=postgres ``` Then from the controller, confirming both that it listens on the LAN and that the Vault password took: ```sh psql "postgres://postgres:$PGPASSWORD@192.168.50.54:5432/postgres" -c 'SELECT version();' ``` ## 6. Node backups Replication covers a dead node, not a dropped table. `roles/pve_backup` writes vzdump archives to the NAS: ```sh ansible-playbook playbooks/pve_host.yml ``` ## Growing the disk Everything lives on the rootfs — that is what makes a replication snapshot atomic (`src/shared/postgres/terraform/README.md`), so there is no second volume to add when space runs short. Raise `disk_size` on the `postgres` module in `build/config/terraform/main.tf` and apply. Check the pool has the room first. Terraform will not: ZFS lets an apply overcommit the pool happily, and the failure surfaces later as a write inside the guest hitting ENOSPC. ```sh ssh root@turtle-proxmox-01 'zpool list AppData; zfs list -o name,used,avail,refquota -r AppData' ssh root@turtle-proxmox-02 'zpool list AppData' # the target needs the room too ``` Then: ```sh cd build/config/terraform terraform plan # expect an in-place update to disk.size, NOT a replacement terraform apply ``` A **replacement** in that plan means something other than the size changed — `prevent_destroy` will refuse it, which is the point. Never work around it here; replacing this container destroys every database on it. The resize itself is a `pct resize` of the rootfs, which on ZFS-backed storage is a refquota change rather than a partition operation. It applies to a running guest, ships no data, and needs nothing done inside the container afterwards — no `resize2fs`, no Postgres restart. Confirm from inside: ```sh ssh root@192.168.50.54 'df -h /' ``` Two things the larger number does *not* change: replication traffic, which is a function of what actually gets written rather than of the quota, and vzdump archive size, which covers used data only. Both track the databases, not the headroom. Shrinking is not available — it is a replacement, and `prevent_destroy` blocks it — so overshoot rather than raising this every few months. ## Failing over Replication makes the far copy a volume, not a running guest, so failover is deliberate. Planned, with both nodes up, is an ordinary migration — fast, because replication means only the delta has to ship: ```sh ssh root@turtle-proxmox-01 'pct migrate 161 turtle-proxmox-02 --restart' ``` Unplanned, with the source node gone, means telling Proxmox to run the guest from the replicated volume, and it loses every transaction committed since the last successful send — up to `replication_schedule`, five minutes. Check what you are about to accept first: ```sh ssh root@turtle-proxmox-02 'pvesr status' # look at "Last Sync" ``` Automating this is what a HA resource would add, and it is deliberately not configured: automatic failover on a two-node cluster with no third vote is a good way to get both nodes deciding they are the survivor. ## Afterwards: repointing clients The address is written down in several files that cannot discover it. The authoritative copy is `ip_address` on the `postgres` module in `build/config/terraform/main.tf`, published as the `postgres_address` output. | Where | What | |---|---| | `inventory/host_vars/.yml` | `db.provision_host` on each app that declares one | | `src//ansible/proxmox/vars.yml` | that app's own `DB_HOST` | | `src//ansible/unraid/vars.yml` | ditto, for anything still on Unraid — no `unraid_shared` network alias reaches this host | Apps on the k3s cluster keep using the in-cluster CloudNativePG instance and are unaffected. ## Migrating an existing database onto it Per database, not `pg_dumpall` — the roles are recreated by the `db:` block on each app's own entry, so only the data needs moving: ```sh pg_dump -h -U postgres -Fc forgejo > forgejo.dump # create the role and database by deploying the app once with its `db:` block, # then: pg_restore -h 192.168.50.54 -U postgres -d forgejo --no-owner --role=forgejo forgejo.dump ``` `--no-owner --role=` rather than a straight restore: the dump carries ownership from the old cluster, and the role provisioning in `lxc_app` / `compose_stack` has already created the owner here with a Vault-managed password. Restoring ownership from the dump would fight it. Stop the application before dumping. A dump of a live database is consistent as of its start, so anything written during it is silently absent from the restore.