scrabble-game/deploy/README.md

deploy

The full Scrabble contour: backend + gateway + the static landing + Postgres + the Telegram validator + bot (the bot with a VPN sidecar) + the observability stack (OTel Collector → Prometheus + Tempo → Grafana), fronted by a caddy that owns a single /_gm Basic-Auth (the admin console + Grafana). Topology and the decision record are in ../docs/ARCHITECTURE.md §13; this file is the operational reference for every environment variable.

Services

Service	Image	Role
caddy	caddy:2-alpine	Edge proxy (alias scrabble on edge): single /_gm Basic-Auth → admin console + Grafana; /app/, /telegram/ + the Connect path → gateway; the catch-all (incl. /) → landing. TLS per CADDY_SITE_ADDRESS.
gateway	built (gateway/Dockerfile, target gateway)	Public edge; serves the embedded game SPA at /app/ + /telegram/; Connect-RPC edge. / redirects to /app/.
landing	built (gateway/Dockerfile, target landing)	Static landing page at / (caddy:2-alpine + the shared Vite build, deploy/landing/Caddyfile); absorbs stray public paths.
backend	built (backend/Dockerfile)	Domain service; bakes in the DAWG dictionaries; runs migrations at boot.
postgres	postgres:17-alpine	Database (named volume, pg_isready healthcheck).
validator	built (platform/telegram/Dockerfile, target validator)	Telegram HMAC validator (no VPN, no Bot API); internal gRPC at validator:9091. Game login depends only on this.
vpn + bot	sidecar + built (platform/telegram/Dockerfile, target bot)	Telegram bot, gated to the telegram-local profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at gateway:9443. The test contour activates the profile; the prod main host omits it and runs the bot standalone on its own host (docker-compose.bot.yml, no VPN — native Bot API egress).
otelcol	otel/opentelemetry-collector-contrib	OTLP/gRPC :4317 → Prometheus scrape (:9464) + Tempo.
prometheus	prom/prometheus	Metrics, 15d retention (7d in prod).
tempo	grafana/tempo	Traces, 72h retention.
grafana	grafana/grafana	Dashboards (provisioned), anonymous-admin behind caddy's /_gm/grafana.
node_exporter	quay.io/prometheus/node-exporter	Host CPU/memory/disk metrics (Prometheus job node); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB).

Networking: inter-service traffic is on the private internal network (project-scoped DNS); only caddy joins the shared external edge network so the host caddy can reach it at scrabble:80. edge must already exist on the host (docker network create edge).

Run it

Locally — copy the template, fill the required values, bring it up:

cp deploy/.env.example deploy/.env     # then edit deploy/.env
docker network create edge             # once, if it does not exist
cd deploy && docker compose up -d --build

In CI (the test contour) — .gitea/workflows/ci.yaml's deploy job maps the Gitea TEST_-prefixed secrets/variables onto the unprefixed names below and runs docker compose up -d --build on the runner host. The prod deploy maps the PROD_ set the same way. So a Gitea secret named TEST_POSTGRES_PASSWORD feeds the compose's POSTGRES_PASSWORD, etc.

The deploy job also seeds the config files (caddy, otelcol, prometheus, tempo, grafana) to a stable host path ($HOME/.scrabble-deploy) and sets SCRABBLE_CONFIG_DIR to it before up. The runner's checkout is an ephemeral act workspace that is removed after the job — binding config straight from it would dangle the mounts in the long-lived containers (Grafana would log no such file or directory). Locally SCRABBLE_CONFIG_DIR defaults to ., so the compose binds from this directory.

Required variables

docker compose aborts immediately if any of these is unset (they use :?):

Variable	Gitea kind	Purpose
POSTGRES_PASSWORD	secret	Postgres password (also embedded in BACKEND_POSTGRES_DSN).
GM_BASICAUTH_HASH	secret	bcrypt hash gating /_gm (admin console + Grafana). Generate with docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'.
TELEGRAM_MINIAPP_URL	variable	The Mini App URL the bot hands out in deep links / buttons.

Plus the bot token — TELEGRAM_BOT_TOKEN (secret), shared by the validator (HMAC secret) and the bot (Bot API). It defaults to empty in compose, but both fail at boot when it is empty.

Conditionally — AWG_CONF (secret): the AmneziaWG config for the VPN sidecar, needed only when the telegram-local profile runs (the test contour and local runs with the bot). It is not :?-guarded — compose interpolates profiled-out services too, so the prod main host (no VPN) must not require it. It must not contain a DNS= line — that hijacks the shared netns's resolv.conf and breaks the bot resolving otelcol / gateway; without it Docker's resolver handles otelcol, gateway and api.telegram.org.

Optional variables (with defaults)

Variable	Gitea kind	Default	Purpose
POSTGRES_DB	variable	scrabble	Database name.
POSTGRES_USER	variable	scrabble	Database user.
DICT_VERSION	variable	v1.3.0	scrabble-dictionary release tag baked into the backend image as the seed for a fresh volume (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded .seed_version marker wins — the seed-drift guard, ARCHITECTURE.md §5). Set per contour as TEST_/PROD_DICT_VERSION.
LOG_LEVEL	variable	info	Shared log level for backend / gateway / validator / bot (debug|info|warn|error).
CADDY_SITE_ADDRESS	variable	:80	Caddy site address. Test: :80 (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME.
GM_BASICAUTH_USER	variable	gm	Username for the /_gm Basic-Auth.
GRAFANA_ROOT_URL	variable	/_gm/grafana/	Grafana root URL (sub-path serving). Set the full https://<domain>/_gm/grafana/ behind a real domain.
GRAFANA_ADMIN_PASSWORD	secret	admin	Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway.
TELEGRAM_GAME_CHANNEL_ID	variable	(empty)	The bot's game-channel id; empty/0 disables channel posts.
TELEGRAM_TEST_ENV	pinned	false	true routes the bot through Telegram's test environment (.../bot<token>/test/METHOD). The CI test contour pins this to true in ci.yaml (the contour is the test environment) — it is not a Gitea variable. Set it in .env for a local run; prod leaves it false.
TELEGRAM_API_BASE_URL	variable	(empty)	Override the Bot API host (a mock/self-hosted server); empty = https://api.telegram.org.
VITE_TELEGRAM_BOT_ID	variable	(empty)	UI build-arg: numeric bot id for the web Login Widget.
VITE_TELEGRAM_LINK	variable	(empty)	UI build-arg: friend-invite Mini App link (full URL, https://t.me/<bot>/<app> — <app> is the Mini App short name from BotFather).
VITE_TELEGRAM_GAME_CHANNEL_NAME	variable	(empty)	UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. https://t.me/Erudit_Game).
VITE_GATEWAY_URL	variable	(empty)	UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy).

The five VITE_* are build-args baked into the gateway and landing images at build time (both targets share one UI build stage — keep the args identical so it is built once), so changing them requires a rebuild (--build), not just a restart.

Fixed internal wiring (not operator-set)

These are hard-wired in docker-compose.yml (no ${...}), pointing the services at each other on the internal network — listed here so they are not mistaken for missing config: BACKEND_POSTGRES_DSN (→ postgres, search_path=backend), GATEWAY_BACKEND_HTTP_URL/_GRPC_ADDR (→ backend), GATEWAY_VALIDATOR_ADDR (→ validator:9091), BACKEND_CONNECTOR_ADDR (→ the gateway bot-link relay gateway:9092), the bot's TELEGRAM_GATEWAY_ADDR (→ gateway:9443, mTLS) with the GATEWAY_BOTLINK_* / TELEGRAM_BOTLINK_* cert paths under /certs (the mTLS material is generated by deploy/gen-certs.sh, gitignored, regenerated each deploy), and all services' *_OTEL_*_EXPORTER=otlp → OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317 (_INSECURE=true). The bot shares the VPN sidecar's netns: routing to the collector's / gateway's internal IP is fine (connected route), but its AWG_CONF must not set a DNS= directive — that hijacks resolv.conf and breaks resolving otelcol / gateway ("produced zero addresses"); without it the netns uses Docker's resolver, which resolves otelcol, gateway and api.telegram.org. GATEWAY_ADMIN_* is intentionally unset — caddy owns /_gm in the contour.

Bumping the dictionary version

The dictionary ships as a versioned release artifact (scrabble-dawg-vX.Y.Z.tar.gz) from scrabble-dictionary. The tag is a build-time input with no default in the images, so it is set in exactly two places to move the whole stack — change both to a new release:

1. CI tests — .gitea/workflows/ci.yaml env.DICT_VERSION (the unit/integration jobs download that dawg).
2. Deploy seed — the Gitea repo variables TEST_DICT_VERSION / PROD_DICT_VERSION (the tag the deploy bakes into a fresh volume's image; the deploy job feeds it to compose as DICT_VERSION).

For local builds set DICT_VERSION in deploy/.env (template: .env.example); a bare docker build needs --build-arg DICT_VERSION=vX.Y.Z. The Dockerfiles and compose carry no default — a missing value fails loudly instead of baking a stale tag.

Bumping the seed is a no-op on a live volume (the .seed_version marker wins — the seed-drift guard). A running contour/prod moves to a new release through the admin console /_gm/dictionary (upload the tarball, preview the per-variant diff, confirm); in-flight games keep their pinned version, new games use the new one (ARCHITECTURE.md §5).

Production rollout

Prod runs on two hosts (main = full stack + ACME on the domain; tg = the bot only, native Bot API, no VPN), one-time provisioned by ansible/ (docker, a non-sudo deploy user holding the CI key, key-only sshd, default-deny ufw, fail2ban). Re-run ansible/ after a host resize — it is idempotent.

To roll out: merge development → master (CI green), then run the prod-deploy workflow manually (Gitea → Actions → prod-deploy → run from master, input confirm=deploy). It builds + pushes the images to the registry, ships the compose/config/certs/env over SSH, deploys the main host with prod-deploy.sh (rolling, health-gated, auto-rollback to the previous tag; caddy is force-recreated on its roll so a bind-mounted Caddyfile change applies — its image is pinned and admin is off, so neither a new tag nor a hot reload would pick it up), then the bot host, then probes the public site. After master is green this workflow is the only thing that touches prod — nothing auto-deploys there. It runs four visible jobs: build → deploy-main → deploy-bot → verify (the per-service rolling shows in the deploy-main log).

Versioning. Each release is a git tag vX.Y.Z on master; the deploy stamps git describe --tags into every image tag, every binary (-ldflags → pkg/version → the service.version telemetry attribute) and the SPA About screen. Tag the release before running the deploy:

git tag -a v1.0.0 -m v1.0.0 && git push origin v1.0.0

Manual rollback (any time after a successful deploy). Run the prod-rollback workflow (Gitea → Actions → prod-rollback, confirm=rollback). Leave target_version blank to roll back to the previously deployed version (read from the host's PREVIOUS_TAG), or set it to a release tag from the Releases page. It re-deploys that already-published image rolling + health-gated — no rebuild, no DB migration (image rollback is DB-safe under the expand-contract rule). The registry keeps every release tag, so any prior release is reachable.

Migrations must be expand-contract (backward-compatible; goose is forward-only): the automatic rollback is image-only and never restores the DB. A deploy that changes backend/internal/postgres/migrations/ opens a maintenance window — the backend (sole writer) is stopped for a consistent pg_dump into /opt/scrabble/dumps before the new backend migrates. Manual DB restore (only if a migration was destructive): docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA backend CASCADE', then pipe the dump into the same psql, and redeploy the matching old tag.

bot-link cert rotation: regenerate (deploy/gen-certs.sh /tmp/c --force), reset the five PROD_BOTLINK_* secrets from /tmp/c, and re-run the workflow — both hosts redeploy together with the fresh CA.

Sizing / monitoring: the main host launches undersized (2 vCPU / 1.9 GiB); the prod overlay trims limits + GOMAXPROCS=2 + 7d Prometheus retention, and node_exporter feeds host memory to Grafana (/_gm/grafana/). Watch host memory and resize at Selectel when players arrive.

PROD_ Gitea set (mirrors TEST_, mapped onto the unprefixed names above) — secrets: PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN, TELEGRAM_PROMO_BOT_TOKEN, REGISTRY_PASSWORD, SSH_KEY, SSH_KNOWN_HOSTS, BOTLINK_CA, BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT, BOTLINK_BOT_KEY}; variables: PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER, GRAFANA_ROOT_URL, LOG_LEVEL, DICT_VERSION, TELEGRAM_MINIAPP_URL, TELEGRAM_GAME_CHANNEL_ID, TELEGRAM_CHAT_ID, TELEGRAM_BOT_USERNAME, VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK, VITE_TELEGRAM_GAME_CHANNEL_NAME}.

Host-side setup (outside this repo)

• edge network must exist on the host (docker network create edge).
• Host caddy route <domain> → scrabble:80 (the in-compose caddy serves HTTP in the test contour; the host caddy terminates TLS). Not needed on prod, where the contour caddy owns TLS (set CADDY_SITE_ADDRESS to the domain).
• Branch protection requires the single status check CI / gate. The unit / integration / ui jobs are path-conditional (they skip when their code did not change), and the always-running gate job aggregates them (passing when each succeeded or was skipped), so a skipped job never blocks a merge. See ../CLAUDE.md "Branching & CI".