homelab/docs/forgejo-proxmox.md

# Bringing up Forgejo on Proxmox

A run-once bootstrap, same shape as `postgres-proxmox.md` and deliberately
downstream of it — Forgejo's database lives on the shared Postgres that
runbook creates, so none of this works until that one is finished.

The end state: an LXC on the `AppData` pool of **turtle-proxmox-02** (VMID
160, 192.168.50.52), running Forgejo natively under systemd, with its database
on the shared Postgres LXC at 192.168.50.54 and a nightly `pg_dump` landing on
its own disk in time for the node's 02:00 vzdump.

## Why turtle-proxmox-02

Nothing forces it. Postgres is on `-01`, so putting Forgejo on `-02` splits
the two guests across the cluster: a node going down takes one of them with
it rather than both. Note what that does *not* buy — Forgejo with its database
unreachable is not a working forge, so this is about not losing both
filesystems at once, not about staying up.

It is set as a literal `node_name` on the `forgejo` module in
`build/config/terraform/main.tf`, not from `var.proxmox_node`. **Change it
only before the container exists**: the provider treats `node_name` as a
replacement, and `prevent_destroy` in `src/forgejo/terraform/main.tf` turns
that into a failed plan. Moving a live Forgejo is a Proxmox migration
(`pct migrate`) followed by editing the literal to match, not a
`terraform apply`.

## 1. Secrets

`homelab/forgejo` needs five keys. Four of them may already exist from the
Unraid stack; **`LFS_JWT_SECRET` almost certainly does not** — the Compose
deployment let Forgejo generate it, and the native install renders `app.ini`
in full with `INSTALL_LOCK = true`, so there is no first boot for Forgejo to
invent one on.

```sh
vault kv get kv/homelab/forgejo
```

If `LFS_JWT_SECRET` is missing, generate one and patch it in without
disturbing the others:

```sh
vault kv patch kv/homelab/forgejo \
  LFS_JWT_SECRET="$(docker run --rm codeberg.org/forgejo/forgejo:10 forgejo generate secret)"
```

Full key table in `vault-secrets.md`. `install.yml` asserts on all five before
it touches the container, so a missing one fails on the first task with a
message naming it rather than three tasks later on an undefined variable.

## 2. The container template

The template must be present on **turtle-proxmox-02**, not just on `-01` where
Postgres was built. If `StorageOne` is not shared across the cluster this is a
separate download:

```sh
ssh root@turtle-proxmox-02 'pveam list StorageOne'
# if the ubuntu-26.04 template is absent:
ssh root@turtle-proxmox-02 'pveam update && pveam download StorageOne ubuntu-26.04-standard_26.04-1_amd64.tar.zst'
```

Whatever is there has to match `lxc_template_file_id` exactly — the filename
moves with each point release, which is why that variable has no default.

## 3. Terraform

State is on the CNPG cluster, so the Pis have to be up (see
`postgres-proxmox.md` § "Why this order").

```sh
cd build/config/terraform
terraform init -backend-config=...        # as per postgres-proxmox.md
terraform plan -out=deploy.plan -target=module.forgejo
terraform apply deploy.plan
```

Read the plan before applying. It should be **one resource to add** and
nothing to change or destroy; anything touching `module.postgres` — in
particular a *replacement* — means stop and work out why, because that guest
is live.

Note `deploy.plan` embeds the state and every input variable, `sensitive` ones
included, so it holds the Proxmox API token in the clear. `.gitignore` covers
`*.plan`; delete it once applied rather than leaving it in the tree.

Confirm the guest exists and Ansible can see it:

```sh
ssh root@turtle-proxmox-02 'pct list | grep 160'
cd ../ansible && ansible-inventory --list --yaml proxmox_guests | grep -A2 forgejo
```

If it is absent from the inventory but present in `pct list`, the `terraform`
tag is missing — that filter is what tells this repo's guests from hand-made
ones, and a guest without it is silently never deployed to.

## 4. Install Forgejo into it

```sh
cd build/config/ansible
ansible-playbook playbooks/proxmox.yml -e only_stacks=forgejo
```

This provisions the `forgejo` database and role on 192.168.50.54, then
installs the binary, `app.ini`, the systemd unit and the dump timer. The
database step runs from **your controller**, not from inside the container
(`delegate_to: localhost`), so 192.168.50.54:5432 has to be reachable from
wherever you run this.

Check:

```sh
ssh root@192.168.50.52 'systemctl is-active forgejo && systemctl list-timers forgejo-dbdump.timer'
curl -sI http://192.168.50.52:3000/ | head -1
```

## 4b. The first admin user

A fresh install has **no way in**: `INSTALL_LOCK = true` skips the setup
wizard, which is where the first admin would normally be created, and
`DISABLE_REGISTRATION = true` means you cannot self-register either (both in
`src/forgejo/ansible/proxmox/templates/app.ini.j2`). Nothing in `install.yml`
creates a user, so the account has to be made from the CLI inside the
container:

```sh
ssh root@192.168.50.52

sudo -u git forgejo admin user create   --admin   --username <you>   --email <you>@turtlesystems.co.uk   --password '<strong-password>'   --config /etc/forgejo/app.ini   --work-path /var/lib/forgejo
```

Three things that are easy to get wrong:

- **`sudo -u git`, not root.** `RUN_USER` is `git`; running the CLI as root
  leaves root-owned files under `/var/lib/forgejo` that Forgejo then cannot
  write. `app.ini` is `0640 root:git`, so the `git` user can read it.
- **`--config` is not optional.** Forgejo looks for `custom/conf/app.ini`
  under the work path by default; this install puts it at
  `/etc/forgejo/app.ini` (`FORGEJO_CONFIG_DIR` in `vars.yml`). Without the
  flag the CLI reads a config that isn't there and never reaches the
  database.
- Recent versions default `--must-change-password` to true, so expect a
  forced change on first web login. Pass `--must-change-password=false` to
  skip it. Confirm the flags for the pinned `FORGEJO_RELEASE` with
  `forgejo admin user create --help` rather than assuming.

Then log in at `http://192.168.50.52:3000/user/login` — by IP, because
`ROOT_URL` is `https://git.turtlesystems.uk` and nothing resolves there until
the cutover in step 7. The login form works over the IP; some links and
redirects Forgejo renders will point at the not-yet-live hostname.

**Skip this step if you are restoring a database in step 6.** Users live in
the database, so a dump brings its own admin back and an account created here
is overwritten by the restore. Create one only if this is genuinely an empty
forge — or after the restore, if the dump turns out to have no usable admin.

## 5. Node backups

VMID 160 is on `-02`, so it is `-02`'s vzdump job that covers it.
`pve_backup_all: true` in `group_vars/proxmox_nodes.yml` means no edit is
needed — but the job has to actually exist on that node:

```sh
ansible-playbook playbooks/pve_host.yml --limit turtle-proxmox-02
ssh root@turtle-proxmox-02 'cat /etc/pve/jobs.cfg'
```

The 01:30 dump and the 02:00 vzdump are a pair. Move one, move the other —
see `src/forgejo/ansible/proxmox/README.md`.

## 6. Repository data

**Ansible does not move any of this.** A fresh install is a working, empty
forge; the repositories are a separate restore into
`/var/lib/forgejo/data/forgejo-repositories`.

The original instructions here were an `rsync` off nas2, which no longer
exists as a machine to read from — whatever repository data survives has to
come from wherever it went when that host was retired. After restoring:

```sh
ssh root@192.168.50.52 'chown -R git:git /var/lib/forgejo/data && systemctl restart forgejo'
```

Restore the database the same way, from whatever dump you have, before
starting Forgejo against it — an empty database with populated repositories
gives back every file and no issues, pull requests, users or permissions.

## 7. Cutover

Last, once the above is verified:

- Point `git.turtlesystems.uk` at whatever terminates HTTPS for it — the Caddy
  reverse proxy at 192.168.50.51, which proxies through to this container's
  `FORGEJO_HTTP_PORT`.
- Point `src.turtlesystems.uk` at this container (192.168.50.52). This is the
  name in `FORGEJO_SSH_DOMAIN`, and the one SSH clone URLs use.
- **SSH clone URLs move from port 2222 to 22, and onto their own hostname.**
  The Unraid stack published 2222 to dodge the host's own sshd; this container
  has its own IP and uses its own sshd, with Forgejo managing the `git` user's
  `authorized_keys`. Every existing remote needs editing — there is no
  redirect for this.

  The two hostnames are not redundancy. An SSH client sends a version banner
  and nothing identifying the host it meant — there is no equivalent of TLS
  SNI — so a reverse proxy cannot route SSH by name the way it routes HTTP.
  The web name therefore has to point at the proxy and the SSH name at
  Forgejo. Collapsing them into one would mean putting a layer-4 TCP proxy on
  the proxy's port 22 (Caddy can, via the third-party `caddy-l4` module, at
  the cost of a custom binary and of losing client IPs on SSH), not
  configuring the hostname more cleverly.
- The Forgejo Actions runner registration is tied to the instance. Re-register
  it against the new host, or CI stops running (`.forgejo/workflows/`).

## 8. Sign-in through Authentik (optional)

Forgejo supports OIDC, and `app.ini` is already set up for it — but only the
*policy* half. The provider itself cannot be configured from `app.ini`:
Forgejo keeps authentication sources in its **database**. That makes this the
one part of Forgejo's configuration Ansible does not own, and the reason it is
a runbook step rather than a task in `install.yml`. It is run once and then
carried forward by the nightly `pg_dump`, the same as every other row in that
database.

**Do this after step 7, not before.** The redirect URI has to match `ROOT_URL`
(`https://git.turtlesystems.uk`), so DNS and TLS must already be live.
Configuring it against `http://192.168.50.52:3000` means doing it twice.

### On the Authentik side

Create an OAuth2/OIDC **Provider** plus an **Application** for it, as for any
other app on `auth.turtlesystems.uk`. The redirect URI is:

```
https://git.turtlesystems.uk/user/oauth2/authentik/callback
```

The last path segment is the *name of the auth source in Forgejo*, not a fixed
string — it has to match the `--name` below. Store the generated client secret
in Vault (`vault-secrets.md` → `OIDC_CLIENT_SECRET`).

### On the Forgejo side

```sh
ssh root@192.168.50.52

sudo -u git forgejo admin auth add-oauth   --name authentik   --provider openidConnect   --key <client-id>   --secret <client-secret>   --auto-discover-url https://auth.turtlesystems.uk/application/o/forgejo/.well-known/openid-configuration   --config /etc/forgejo/app.ini   --work-path /var/lib/forgejo
```

Same `sudo -u git` and `--config` requirements as step 4b, for the same
reasons. Verify the flag names against the pinned `FORGEJO_RELEASE` with
`forgejo admin auth add-oauth --help` before running — and list what exists
afterwards with `forgejo admin auth list`, which is also how you find the `id`
for `update-oauth` if the secret is ever rotated.

No restart is needed: the source is a database row, not a file Forgejo reads
at boot.

### What the app.ini side already does

Set in `src/forgejo/ansible/proxmox/templates/app.ini.j2`, so a redeploy keeps
them — the comments there carry the detail:

| Setting | Effect |
|---|---|
| `DISABLE_REGISTRATION = false` | It blocks OIDC auto-registration too, not just the local signup form. Left `true`, Authentik logins authenticate and are then refused an account. |
| `ALLOW_ONLY_EXTERNAL_REGISTRATION = true` | Restores "no self-service signup" without blocking Authentik. |
| `ENABLE_AUTO_REGISTRATION = true` | Creates the Forgejo account on first successful SSO login. |
| `ACCOUNT_LINKING = auto` | Attaches the OIDC identity to an existing local account with the same email. |

`ACCOUNT_LINKING` is the one that matters on the first login: if the
break-glass admin from step 4b uses the same email address as your Authentik
account, signing in through Authentik lands you *in that admin account*. With
linking off you would arrive as a second, unprivileged user and have to
promote it from the CLI.

### Keep the local admin

Do not delete the step 4b account or convert it to SSO-only. The login path is
now Forgejo → Authentik → CloudNativePG → a healthy k3s cluster, and any of
those failing takes SSO down with it — including the cases you would most want
to log into the forge to investigate. This is the same reasoning
`roles/k3s_traefik` uses for keeping the port-forward-only dashboard route
alive alongside the Authentik-published one.

### Git over HTTPS

Users who arrive through Authentik have no Forgejo password, so HTTPS clones
need a personal access token. SSH keys are unaffected — and since step 7 put
git-over-SSH on port 22 with Forgejo managing the `git` user's
`authorized_keys`, SSH is the smoother default to point people at.

## Rolling back

There is no `state: absent` for `lxc_app` — removal would need a per-app
`uninstall.yml` and Forgejo has none. To back out: stop the service
(`systemctl stop forgejo`), point DNS back at whatever was serving before, and
leave the container in place. Destroying it needs the `prevent_destroy` block
removed by hand first, which is deliberate.