homelab/docs/forgejo-proxmox.md

# Bringing up Forgejo on Proxmox

A run-once bootstrap, same shape as `postgres-proxmox.md` and deliberately
downstream of it — Forgejo's database lives on the shared Postgres that
runbook creates, so none of this works until that one is finished.

The end state: an LXC on the `AppData` pool of **turtle-proxmox-02** (VMID
160, 192.168.50.52), running Forgejo natively under systemd, with its database
on the shared Postgres LXC at 192.168.50.54 and a nightly `pg_dump` landing on
its own disk in time for the node's 02:00 vzdump.

## Why turtle-proxmox-02

Nothing forces it. Postgres is on `-01`, so putting Forgejo on `-02` splits
the two guests across the cluster: a node going down takes one of them with
it rather than both. Note what that does *not* buy — Forgejo with its database
unreachable is not a working forge, so this is about not losing both
filesystems at once, not about staying up.

It is set as a literal `node_name` on the `forgejo` module in
`build/config/terraform/main.tf`, not from `var.proxmox_node`. **Change it
only before the container exists**: the provider treats `node_name` as a
replacement, and `prevent_destroy` in `src/forgejo/terraform/main.tf` turns
that into a failed plan. Moving a live Forgejo is a Proxmox migration
(`pct migrate`) followed by editing the literal to match, not a
`terraform apply`.

## 1. Secrets

`homelab/forgejo` needs five keys. Four of them may already exist from the
Unraid stack; **`LFS_JWT_SECRET` almost certainly does not** — the Compose
deployment let Forgejo generate it, and the native install renders `app.ini`
in full with `INSTALL_LOCK = true`, so there is no first boot for Forgejo to
invent one on.

```sh
vault kv get kv/homelab/forgejo
```

If `LFS_JWT_SECRET` is missing, generate one and patch it in without
disturbing the others:

```sh
vault kv patch kv/homelab/forgejo \
  LFS_JWT_SECRET="$(docker run --rm codeberg.org/forgejo/forgejo:10 forgejo generate secret)"
```

Full key table in `vault-secrets.md`. `install.yml` asserts on all five before
it touches the container, so a missing one fails on the first task with a
message naming it rather than three tasks later on an undefined variable.

## 2. The container template

The template must be present on **turtle-proxmox-02**, not just on `-01` where
Postgres was built. If `StorageOne` is not shared across the cluster this is a
separate download:

```sh
ssh root@turtle-proxmox-02 'pveam list StorageOne'
# if the ubuntu-26.04 template is absent:
ssh root@turtle-proxmox-02 'pveam update && pveam download StorageOne ubuntu-26.04-standard_26.04-1_amd64.tar.zst'
```

Whatever is there has to match `lxc_template_file_id` exactly — the filename
moves with each point release, which is why that variable has no default.

## 3. Terraform

State is on the CNPG cluster, so the Pis have to be up (see
`postgres-proxmox.md` § "Why this order").

```sh
cd build/config/terraform
terraform init -backend-config=...        # as per postgres-proxmox.md
terraform plan -out=deploy.plan -target=module.forgejo
terraform apply deploy.plan
```

Read the plan before applying. It should be **one resource to add** and
nothing to change or destroy; anything touching `module.postgres` — in
particular a *replacement* — means stop and work out why, because that guest
is live.

Note `deploy.plan` embeds the state and every input variable, `sensitive` ones
included, so it holds the Proxmox API token in the clear. `.gitignore` covers
`*.plan`; delete it once applied rather than leaving it in the tree.

Confirm the guest exists and Ansible can see it:

```sh
ssh root@turtle-proxmox-02 'pct list | grep 160'
cd ../ansible && ansible-inventory --list --yaml proxmox_guests | grep -A2 forgejo
```

If it is absent from the inventory but present in `pct list`, the `terraform`
tag is missing — that filter is what tells this repo's guests from hand-made
ones, and a guest without it is silently never deployed to.

## 4. Install Forgejo into it

```sh
cd build/config/ansible
ansible-playbook playbooks/proxmox.yml -e only_stacks=forgejo
```

This provisions the `forgejo` database and role on 192.168.50.54, then
installs the binary, `app.ini`, the systemd unit and the dump timer. The
database step runs from **your controller**, not from inside the container
(`delegate_to: localhost`), so 192.168.50.54:5432 has to be reachable from
wherever you run this.

Check:

```sh
ssh root@192.168.50.52 'systemctl is-active forgejo && systemctl list-timers forgejo-dbdump.timer'
curl -sI http://192.168.50.52:3000/ | head -1
```

## 4b. The first admin user

A fresh install has **no way in**: `INSTALL_LOCK = true` skips the setup
wizard, which is where the first admin would normally be created, and
`DISABLE_REGISTRATION = true` means you cannot self-register either (both in
`src/forgejo/ansible/proxmox/templates/app.ini.j2`). Nothing in `install.yml`
creates a user, so the account has to be made from the CLI inside the
container:

```sh
ssh root@192.168.50.52

sudo -u git forgejo admin user create   --admin   --username <you>   --email <you>@turtlesystems.co.uk   --password '<strong-password>'   --config /etc/forgejo/app.ini   --work-path /var/lib/forgejo
```

Three things that are easy to get wrong:

- **`sudo -u git`, not root.** `RUN_USER` is `git`; running the CLI as root
  leaves root-owned files under `/var/lib/forgejo` that Forgejo then cannot
  write. `app.ini` is `0640 root:git`, so the `git` user can read it.
- **`--config` is not optional.** Forgejo looks for `custom/conf/app.ini`
  under the work path by default; this install puts it at
  `/etc/forgejo/app.ini` (`FORGEJO_CONFIG_DIR` in `vars.yml`). Without the
  flag the CLI reads a config that isn't there and never reaches the
  database.
- Recent versions default `--must-change-password` to true, so expect a
  forced change on first web login. Pass `--must-change-password=false` to
  skip it. Confirm the flags for the pinned `FORGEJO_RELEASE` with
  `forgejo admin user create --help` rather than assuming.

Then log in at `http://192.168.50.52:3000/user/login` — by IP, because
`ROOT_URL` is `https://git.turtlesystems.uk` and nothing resolves there until
the cutover in step 7. The login form works over the IP; some links and
redirects Forgejo renders will point at the not-yet-live hostname.

**Skip this step if you are restoring a database in step 6.** Users live in
the database, so a dump brings its own admin back and an account created here
is overwritten by the restore. Create one only if this is genuinely an empty
forge — or after the restore, if the dump turns out to have no usable admin.

## 5. Node backups

VMID 160 is on `-02`, so it is `-02`'s vzdump job that covers it.
`pve_backup_all: true` in `group_vars/proxmox_nodes.yml` means no edit is
needed — but the job has to actually exist on that node:

```sh
ansible-playbook playbooks/pve_host.yml --limit turtle-proxmox-02
ssh root@turtle-proxmox-02 'cat /etc/pve/jobs.cfg'
```

The 01:30 dump and the 02:00 vzdump are a pair. Move one, move the other —
see `src/forgejo/ansible/proxmox/README.md`.

## 6. Repository data

**Ansible does not move any of this.** A fresh install is a working, empty
forge; the repositories are a separate restore into
`/var/lib/forgejo/data/forgejo-repositories`.

The original instructions here were an `rsync` off nas2, which no longer
exists as a machine to read from — whatever repository data survives has to
come from wherever it went when that host was retired. After restoring:

```sh
ssh root@192.168.50.52 'chown -R git:git /var/lib/forgejo/data && systemctl restart forgejo'
```

Restore the database the same way, from whatever dump you have, before
starting Forgejo against it — an empty database with populated repositories
gives back every file and no issues, pull requests, users or permissions.

## 7. Cutover

Last, once the above is verified:

- Point `git.turtlesystems.uk` at this container.
- **SSH clone URLs move from port 2222 to 22.** The Unraid stack published
  2222 to dodge the host's own sshd; this container has its own IP and uses
  its own sshd, with Forgejo managing the `git` user's `authorized_keys`.
  Every existing remote needs editing — there is no redirect for this.
- The Forgejo Actions runner registration is tied to the instance. Re-register
  it against the new host, or CI stops running (`.forgejo/workflows/`).

## 8. Sign-in through Authentik (optional)

Forgejo supports OIDC, and `app.ini` is already set up for it — but only the
*policy* half. The provider itself cannot be configured from `app.ini`:
Forgejo keeps authentication sources in its **database**. That makes this the
one part of Forgejo's configuration Ansible does not own, and the reason it is
a runbook step rather than a task in `install.yml`. It is run once and then
carried forward by the nightly `pg_dump`, the same as every other row in that
database.

**Do this after step 7, not before.** The redirect URI has to match `ROOT_URL`
(`https://git.turtlesystems.uk`), so DNS and TLS must already be live.
Configuring it against `http://192.168.50.52:3000` means doing it twice.

### On the Authentik side

Create an OAuth2/OIDC **Provider** plus an **Application** for it, as for any
other app on `auth.turtlesystems.uk`. The redirect URI is:

```
https://git.turtlesystems.uk/user/oauth2/authentik/callback
```

The last path segment is the *name of the auth source in Forgejo*, not a fixed
string — it has to match the `--name` below. Store the generated client secret
in Vault (`vault-secrets.md` → `OIDC_CLIENT_SECRET`).

### On the Forgejo side

```sh
ssh root@192.168.50.52

sudo -u git forgejo admin auth add-oauth   --name authentik   --provider openidConnect   --key <client-id>   --secret <client-secret>   --auto-discover-url https://auth.turtlesystems.uk/application/o/forgejo/.well-known/openid-configuration   --config /etc/forgejo/app.ini   --work-path /var/lib/forgejo
```

Same `sudo -u git` and `--config` requirements as step 4b, for the same
reasons. Verify the flag names against the pinned `FORGEJO_RELEASE` with
`forgejo admin auth add-oauth --help` before running — and list what exists
afterwards with `forgejo admin auth list`, which is also how you find the `id`
for `update-oauth` if the secret is ever rotated.

No restart is needed: the source is a database row, not a file Forgejo reads
at boot.

### What the app.ini side already does

Set in `src/forgejo/ansible/proxmox/templates/app.ini.j2`, so a redeploy keeps
them — the comments there carry the detail:

| Setting | Effect |
|---|---|
| `DISABLE_REGISTRATION = false` | It blocks OIDC auto-registration too, not just the local signup form. Left `true`, Authentik logins authenticate and are then refused an account. |
| `ALLOW_ONLY_EXTERNAL_REGISTRATION = true` | Restores "no self-service signup" without blocking Authentik. |
| `ENABLE_AUTO_REGISTRATION = true` | Creates the Forgejo account on first successful SSO login. |
| `ACCOUNT_LINKING = auto` | Attaches the OIDC identity to an existing local account with the same email. |

`ACCOUNT_LINKING` is the one that matters on the first login: if the
break-glass admin from step 4b uses the same email address as your Authentik
account, signing in through Authentik lands you *in that admin account*. With
linking off you would arrive as a second, unprivileged user and have to
promote it from the CLI.

### Keep the local admin

Do not delete the step 4b account or convert it to SSO-only. The login path is
now Forgejo → Authentik → CloudNativePG → a healthy k3s cluster, and any of
those failing takes SSO down with it — including the cases you would most want
to log into the forge to investigate. This is the same reasoning
`roles/k3s_traefik` uses for keeping the port-forward-only dashboard route
alive alongside the Authentik-published one.

### Git over HTTPS

Users who arrive through Authentik have no Forgejo password, so HTTPS clones
need a personal access token. SSH keys are unaffected — and since step 7 put
git-over-SSH on port 22 with Forgejo managing the `git` user's
`authorized_keys`, SSH is the smoother default to point people at.

## Rolling back

There is no `state: absent` for `lxc_app` — removal would need a per-app
`uninstall.yml` and Forgejo has none. To back out: stop the service
(`systemctl stop forgejo`), point DNS back at whatever was serving before, and
leave the container in place. Destroying it needs the `prevent_destroy` block
removed by hand first, which is deliberate.