scrabble-game/deploy/README.md

# deploy

The full Scrabble contour: `backend` + `gateway` + the static `landing` + Postgres +
the Telegram `validator` + `bot` (the bot with a VPN sidecar) + the observability stack
(OTel Collector → Prometheus + Tempo → Grafana), fronted by a **caddy** that owns a single
`/_gm` Basic-Auth (the admin console + Grafana). Topology and the decision record are in
[`../docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) §13; this file is the
operational reference for **every environment variable**.

## Services

| Service | Image | Role |
| --- | --- | --- |
| `caddy` | `caddy:2-alpine` | Edge proxy (alias `scrabble` on `edge`): single `/_gm` Basic-Auth → admin console + Grafana; `/app/`, `/telegram/` + the Connect path → gateway; the catch-all (incl. `/`) → landing. TLS per `CADDY_SITE_ADDRESS`. |
| `gateway` | built (`gateway/Dockerfile`, target `gateway`) | Public edge; serves the embedded game SPA at `/app/` + `/telegram/`; Connect-RPC edge. `/` redirects to `/app/`. |
| `landing` | built (`gateway/Dockerfile`, target `landing`) | Static landing page at `/` (caddy:2-alpine + the shared Vite build, `deploy/landing/Caddyfile`); absorbs stray public paths. |
| `backend` | built (`backend/Dockerfile`) | Domain service; bakes in the DAWG dictionaries; runs migrations at boot. |
| `postgres` | `postgres:17-alpine` | Database (named volume, `pg_isready` healthcheck). |
| `validator` | built (`platform/telegram/Dockerfile`, target `validator`) | Telegram HMAC validator (no VPN, no Bot API); internal gRPC at `validator:9091`. Game login depends only on this. |
| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot, gated to the **`telegram-local`** profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at `gateway:9443`. The test contour activates the profile; the prod **main** host omits it and runs the bot standalone on its **own host** (`docker-compose.bot.yml`, no VPN — native Bot API egress). |
| `otelcol` | `otel/opentelemetry-collector-contrib` | OTLP/gRPC `:4317` → Prometheus scrape (`:9464`) + Tempo. |
| `prometheus` | `prom/prometheus` | Metrics, 15d retention (7d in prod). |
| `tempo` | `grafana/tempo` | Traces, 72h retention. |
| `grafana` | `grafana/grafana` | Dashboards (provisioned), anonymous-admin behind caddy's `/_gm/grafana`. |
| `node_exporter` | `quay.io/prometheus/node-exporter` | Host CPU/memory/disk metrics (Prometheus job `node`); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB). |

Networking: inter-service traffic is on the private `internal` network
(project-scoped DNS); only `caddy` joins the shared external `edge` network so the
host caddy can reach it at `scrabble:80`. `edge` must already exist on the host
(`docker network create edge`).

## Run it

**Locally** — copy the template, fill the required values, bring it up:

```sh
cp deploy/.env.example deploy/.env     # then edit deploy/.env
docker network create edge             # once, if it does not exist
cd deploy && docker compose up -d --build
```

**In CI** (the test contour) — `.gitea/workflows/ci.yaml`'s `deploy` job maps the
Gitea **`TEST_`-prefixed** secrets/variables onto the unprefixed names below and
runs `docker compose up -d --build` on the runner host. The prod deploy maps the
**`PROD_`** set the same way. So a Gitea secret named `TEST_POSTGRES_PASSWORD`
feeds the compose's `POSTGRES_PASSWORD`, etc.

The deploy job also **seeds the config files** (`caddy`, `otelcol`, `prometheus`,
`tempo`, `grafana`) to a stable host path (`$HOME/.scrabble-deploy`) and sets
`SCRABBLE_CONFIG_DIR` to it before `up`. The runner's checkout is an ephemeral act
workspace that is removed after the job — binding config straight from it would
dangle the mounts in the long-lived containers (Grafana would log
`no such file or directory`). Locally `SCRABBLE_CONFIG_DIR` defaults to `.`, so the
compose binds from this directory.

## Required variables

`docker compose` aborts immediately if any of these is unset (they use `:?`):

| Variable | Gitea kind | Purpose |
| --- | --- | --- |
| `POSTGRES_PASSWORD` | secret | Postgres password (also embedded in `BACKEND_POSTGRES_DSN`). |
| `GM_BASICAUTH_HASH` | secret | bcrypt hash gating `/_gm` (admin console + Grafana). Generate with `docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'`. |
| `TELEGRAM_MINIAPP_URL` | variable | The Mini App URL the bot hands out in deep links / buttons. |

**Plus the bot token** — `TELEGRAM_BOT_TOKEN` (secret), shared by the validator (HMAC
secret) and the bot (Bot API). It defaults to empty in compose, but both **fail at
boot** when it is empty.

**Conditionally — `AWG_CONF`** (secret): the AmneziaWG config for the VPN sidecar, needed
only when the `telegram-local` profile runs (the test contour and local runs with the
bot). It is **not** `:?`-guarded — compose interpolates profiled-out services too, so the
prod main host (no VPN) must not require it. It **must not contain a `DNS=` line** — that
hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`;
without it Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`.

## Optional variables (with defaults)

| Variable | Gitea kind | Default | Purpose |
| --- | --- | --- | --- |
| `POSTGRES_DB` | variable | `scrabble` | Database name. |
| `POSTGRES_USER` | variable | `scrabble` | Database user. |
| `DICT_VERSION` | variable | `v1.3.0` | `scrabble-dictionary` release tag baked into the backend image as the **seed for a fresh volume** (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded `.seed_version` marker wins — the seed-drift guard, ARCHITECTURE.md §5). Set per contour as `TEST_`/`PROD_DICT_VERSION`. |
| `LOG_LEVEL` | variable | `info` | Shared log level for backend / gateway / validator / bot (`debug\|info\|warn\|error`). |
| `CADDY_SITE_ADDRESS` | variable | `:80` | Caddy site address. Test: `:80` (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME. |
| `GM_BASICAUTH_USER` | variable | `gm` | Username for the `/_gm` Basic-Auth. |
| `GRAFANA_ROOT_URL` | variable | `/_gm/grafana/` | Grafana root URL (sub-path serving). Set the full `https://<domain>/_gm/grafana/` behind a real domain. |
| `GRAFANA_ADMIN_PASSWORD` | secret | `admin` | Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway. |
| `TELEGRAM_GAME_CHANNEL_ID` | variable | _(empty)_ | The bot's game-channel id; empty/`0` disables channel posts. |
| `TELEGRAM_TEST_ENV` | _pinned_ | `false` | `true` routes the bot through Telegram's test environment (`.../bot<token>/test/METHOD`). **The CI test contour pins this to `true` in `ci.yaml`** (the contour is the test environment) — it is not a Gitea variable. Set it in `.env` for a local run; prod leaves it `false`. |
| `TELEGRAM_API_BASE_URL` | variable | _(empty)_ | Override the Bot API host (a mock/self-hosted server); empty = `https://api.telegram.org`. |
| `VITE_TELEGRAM_BOT_ID` | variable | _(empty)_ | UI build-arg: numeric bot id for the web Login Widget. |
| `VITE_TELEGRAM_LINK` | variable | _(empty)_ | UI build-arg: friend-invite Mini App link (full URL, `https://t.me/<bot>/<app>` — `<app>` is the Mini App short name from BotFather). |
| `VITE_TELEGRAM_GAME_CHANNEL_NAME` | variable | _(empty)_ | UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. `https://t.me/Erudit_Game`). |
| `VITE_GATEWAY_URL` | variable | _(empty)_ | UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy). |

The five `VITE_*` are **build-args** baked into the gateway and landing images at
build time (both targets share one UI build stage — keep the args identical so it is
built once), so changing them requires a rebuild (`--build`), not just a restart.

## Fixed internal wiring (not operator-set)

These are hard-wired in `docker-compose.yml` (no `${...}`), pointing the services
at each other on the `internal` network — listed here so they are not mistaken for
missing config: `BACKEND_POSTGRES_DSN` (→ `postgres`, `search_path=backend`),
`GATEWAY_BACKEND_HTTP_URL`/`_GRPC_ADDR` (→ `backend`),
`GATEWAY_VALIDATOR_ADDR` (→ `validator:9091`), `BACKEND_CONNECTOR_ADDR` (→ the gateway
bot-link relay `gateway:9092`), the bot's `TELEGRAM_GATEWAY_ADDR` (→ `gateway:9443`,
mTLS) with the `GATEWAY_BOTLINK_*` / `TELEGRAM_BOTLINK_*` cert paths under `/certs` (the
mTLS material is generated by `deploy/gen-certs.sh`, gitignored, regenerated each
deploy), and all services' `*_OTEL_*_EXPORTER=otlp` →
`OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317`
(`_INSECURE=true`). The bot shares the VPN sidecar's netns: routing to the
collector's / gateway's internal IP is fine (connected route), but its `AWG_CONF` must
**not** set a `DNS=` directive — that hijacks resolv.conf and breaks resolving `otelcol`
/ `gateway` ("produced zero addresses"); without it the netns uses Docker's resolver,
which resolves `otelcol`, `gateway` and `api.telegram.org`. `GATEWAY_ADMIN_*` is
intentionally **unset** — caddy owns `/_gm` in the contour.

## Bumping the dictionary version

The dictionary ships as a versioned **release artifact** (`scrabble-dawg-vX.Y.Z.tar.gz`) from
[`scrabble-dictionary`](https://gitea.iliadenisov.ru/developer/scrabble-dictionary). The tag is
a build-time input with **no default** in the images, so it is set in exactly two places to
move the whole stack — change both to a new release:

1. **CI tests** — `.gitea/workflows/ci.yaml` `env.DICT_VERSION` (the unit/integration jobs
   download that dawg).
2. **Deploy seed** — the Gitea repo variables `TEST_DICT_VERSION` / `PROD_DICT_VERSION` (the tag
   the deploy bakes into a **fresh** volume's image; the deploy job feeds it to `compose` as
   `DICT_VERSION`).

For local builds set `DICT_VERSION` in `deploy/.env` (template: `.env.example`); a bare
`docker build` needs `--build-arg DICT_VERSION=vX.Y.Z`. The Dockerfiles and `compose` carry no
default — a missing value fails loudly instead of baking a stale tag.

Bumping the seed is a **no-op on a live volume** (the `.seed_version` marker wins — the
seed-drift guard). A running contour/prod moves to a new release **through the admin console**
`/_gm/dictionary` (upload the tarball, preview the per-variant diff, confirm); in-flight games
keep their pinned version, new games use the new one (ARCHITECTURE.md §5).

## Production rollout

Prod runs on **two hosts** (main = full stack + ACME on the domain; tg = the bot only,
native Bot API, no VPN), one-time provisioned by **[`ansible/`](ansible/)** (docker, a
non-sudo `deploy` user holding the CI key, key-only sshd, default-deny ufw, fail2ban).
Re-run `ansible/` after a host resize — it is idempotent.

**To roll out:** merge `development → master` (CI green), then run the **`prod-deploy`**
workflow manually (Gitea → Actions → prod-deploy → run from `master`, input
`confirm=deploy`). It builds + pushes the images to the registry, ships the
compose/config/certs/env over SSH, deploys the main host with `prod-deploy.sh` (rolling,
health-gated, **auto-rollback to the previous tag**; caddy is force-recreated on its roll so
a bind-mounted `Caddyfile` change applies — its image is pinned and admin is off, so neither a
new tag nor a hot reload would pick it up), then the bot host, then probes the
public site. After `master` is green this workflow is the **only** thing that touches
prod — nothing auto-deploys there. It runs four visible jobs: **build → deploy-main →
deploy-bot → verify** (the per-service rolling shows in the deploy-main log).

**Versioning.** Each release is a git tag `vX.Y.Z` on `master`; the deploy stamps
`git describe --tags` into every image tag, every binary (`-ldflags` → `pkg/version` →
the `service.version` telemetry attribute) and the SPA About screen. Tag the release
before running the deploy:

```sh
git tag -a v1.0.0 -m v1.0.0 && git push origin v1.0.0
```

**Manual rollback** (any time after a successful deploy). Run the **`prod-rollback`**
workflow (Gitea → Actions → prod-rollback, `confirm=rollback`). Leave `target_version`
blank to roll back to the previously deployed version (read from the host's
`PREVIOUS_TAG`), or set it to a release tag from the **Releases** page. It re-deploys
that already-published image rolling + health-gated — no rebuild, no DB migration
(image rollback is DB-safe under the expand-contract rule). The registry keeps every
release tag, so any prior release is reachable.

**Migrations** must be **expand-contract** (backward-compatible; goose is forward-only):
the automatic rollback is image-only and never restores the DB. A deploy that changes
`backend/internal/postgres/migrations/` opens a maintenance window — the backend (sole
writer) is stopped for a consistent `pg_dump` into `/opt/scrabble/dumps` before the new
backend migrates. **Manual DB restore** (only if a migration was destructive):
`docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA backend CASCADE'`,
then pipe the dump into the same `psql`, and redeploy the matching old tag.

**bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy
together with the fresh CA.

**Sizing / monitoring:** the main host launches undersized (2 vCPU / 1.9 GiB); the prod
overlay trims limits + `GOMAXPROCS=2` + 7d Prometheus retention, and `node_exporter` feeds
host memory to Grafana (`/_gm/grafana/`). Watch host memory and resize at Selectel when
players arrive.

**`PROD_` Gitea set** (mirrors `TEST_`, mapped onto the unprefixed names above) — secrets:
`PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN,
TELEGRAM_PROMO_BOT_TOKEN, REGISTRY_PASSWORD, SSH_KEY, SSH_KNOWN_HOSTS, BOTLINK_CA,
BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT, BOTLINK_BOT_KEY}`; variables:
`PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER,
GRAFANA_ROOT_URL, LOG_LEVEL, DICT_VERSION, TELEGRAM_MINIAPP_URL, TELEGRAM_GAME_CHANNEL_ID,
TELEGRAM_CHAT_ID, TELEGRAM_BOT_USERNAME, VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK,
VITE_TELEGRAM_GAME_CHANNEL_NAME}`.

## Host-side setup (outside this repo)

- **`edge` network** must exist on the host (`docker network create edge`).
- **Host caddy** route `<domain> → scrabble:80` (the in-compose caddy serves HTTP
  in the test contour; the host caddy terminates TLS). Not needed on prod, where the
  contour caddy owns TLS (set `CADDY_SITE_ADDRESS` to the domain).
- **Branch protection** requires the single status check `CI / gate`.
  The `unit` / `integration` / `ui` jobs are path-conditional (they skip when their
  code did not change), and the always-running `gate` job aggregates them (passing
  when each succeeded or was skipped), so a skipped job never blocks a merge. See
  [`../CLAUDE.md`](../CLAUDE.md) "Branching & CI".