scrabble-game/deploy

deploy

The full Scrabble contour: backend + gateway + the static landing + Postgres + the Telegram validator + bot (the bot with a VPN sidecar) + the observability stack (OTel Collector → Prometheus + Tempo → Grafana), fronted by a caddy that owns a single /_gm Basic-Auth (the admin console + Grafana). Topology and the decision record are in ../docs/ARCHITECTURE.md §13; this file is the operational reference for every environment variable.

Services

Service	Image	Role
caddy	caddy:2-alpine	Edge proxy (alias scrabble on edge): single /_gm Basic-Auth → admin console + Grafana; /app/, /telegram/ + the Connect path → gateway; the catch-all (incl. /) → landing. TLS per CADDY_SITE_ADDRESS.
gateway	built (gateway/Dockerfile, target gateway)	Public edge; serves the embedded game SPA at /app/ + /telegram/; Connect-RPC edge. / redirects to /app/.
landing	built (gateway/Dockerfile, target landing)	Static landing page at / (caddy:2-alpine + the shared Vite build, deploy/landing/Caddyfile); absorbs stray public paths.
backend	built (backend/Dockerfile)	Domain service; bakes in the DAWG dictionaries; runs migrations at boot.
postgres	postgres:17-alpine	Database (named volume, pg_isready healthcheck).
renderer	built (renderer/Dockerfile)	Finished-game image-render sidecar (Node + skia-canvas running the shared ui/src/lib/gameimage.ts); internal-only HTTP at renderer:8090, called by the backend for the PNG export artifact.
validator	built (platform/telegram/Dockerfile, target validator)	Telegram HMAC validator (no VPN, no Bot API); internal gRPC at validator:9091. Game login depends only on this.
vpn + bot	sidecar + built (platform/telegram/Dockerfile, target bot)	Telegram bot, gated to the telegram-local profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at gateway:9443. The test contour activates the profile; the prod main host omits it and runs the bot standalone on its own host (docker-compose.bot.yml, no VPN — native Bot API egress).
otelcol	otel/opentelemetry-collector-contrib	OTLP/gRPC :4317 → Prometheus scrape (:9464) + Tempo.
prometheus	prom/prometheus	Metrics, 15d retention (7d in prod).
tempo	grafana/tempo	Traces, 72h retention.
grafana	grafana/grafana	Dashboards (provisioned), anonymous-admin behind caddy's /_gm/grafana.
node_exporter	quay.io/prometheus/node-exporter	Host CPU/memory/disk metrics (Prometheus job node); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB).

Networking: inter-service traffic is on the private internal network (project-scoped DNS); only caddy joins the shared external edge network so the host caddy can reach it at scrabble:80. edge must already exist on the host (docker network create edge).

Run it

Locally — copy the template, fill the required values, bring it up:

cp deploy/.env.example deploy/.env     # then edit deploy/.env
docker network create edge             # once, if it does not exist
cd deploy && docker compose up -d --build

In CI (the test contour) — .gitea/workflows/ci.yaml's deploy job maps the Gitea TEST_-prefixed secrets/variables onto the unprefixed names below and runs docker compose up -d --build on the runner host. The prod deploy maps the PROD_ set the same way. So a Gitea secret named TEST_POSTGRES_PASSWORD feeds the compose's POSTGRES_PASSWORD, etc.

Three naming classes in Gitea:

• Per-contour (TEST_/PROD_<NAME>) — values that differ between the contours (bots, hosts, public origin, log level, email senders): the common case below.
• Shared (one unprefixed <NAME>, no prefix) — values identical on every contour, stored once: DICT_VERSION, SMTP_RELAY_HOST/PORT/TLS/USER/PASS, GRAFANA_SMTP_PORT, VITE_VK_APP_LINK, VITE_VK_APP_ID, GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET (one Selectel relay + one pair of VK apps for all contours).
• Derived — not stored at all; the deploy computes them from PUBLIC_BASE_URL (deploy/write-prod-env.sh, and the ci.yaml deploy step): TELEGRAM_MINIAPP_URL (+ /telegram/), GRAFANA_ROOT_URL (+ /_gm/grafana/), VITE_VK_ID_REDIRECT_URL (+ /app/). Set them directly only for a local .env run.

The deploy job also seeds the config files (caddy, otelcol, prometheus, tempo, grafana) to a stable host path ($HOME/.scrabble-deploy) and sets SCRABBLE_CONFIG_DIR to it before up. The runner's checkout is an ephemeral act workspace that is removed after the job — binding config straight from it would dangle the mounts in the long-lived containers (Grafana would log no such file or directory). Locally SCRABBLE_CONFIG_DIR defaults to ., so the compose binds from this directory.

Required variables

docker compose aborts immediately if any of these is unset (they use :?):

Variable	Gitea kind	Purpose
POSTGRES_PASSWORD	secret	Postgres password (also embedded in BACKEND_POSTGRES_DSN).
GM_BASICAUTH_HASH	secret	bcrypt hash gating /_gm (admin console + Grafana). Generate with docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'.
TELEGRAM_MINIAPP_URL	derived	The Mini App URL the bot hands out in deep links / buttons. The deploy derives PUBLIC_BASE_URL + /telegram/; set it directly only for a local run (compose still :?-requires it).
EXPORT_SIGN_KEY	secret	HMAC key signing the public finished-game export download URLs (/dl/*). Generate with openssl rand -base64 32.

Plus the bot token — TELEGRAM_BOT_TOKEN (secret), shared by the validator (HMAC secret) and the bot (Bot API). It defaults to empty in compose, but both fail at boot when it is empty.

Conditionally — AWG_CONF (secret): the AmneziaWG config for the VPN sidecar, needed only when the telegram-local profile runs (the test contour and local runs with the bot). It is not :?-guarded — compose interpolates profiled-out services too, so the prod main host (no VPN) must not require it. It must not contain a DNS= line — that hijacks the shared netns's resolv.conf and breaks the bot resolving otelcol / gateway; without it Docker's resolver handles otelcol, gateway and api.telegram.org.

Optional variables (with defaults)

Variable	Gitea kind	Default	Purpose
POSTGRES_DB	variable	scrabble	Database name.
POSTGRES_USER	variable	scrabble	Database user.
DICT_VERSION	variable (shared)	v1.3.1	scrabble-dictionary release tag baked into the backend image as the seed for a fresh volume (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded .seed_version marker wins — the seed-drift guard, ARCHITECTURE.md §5). One shared unprefixed Gitea variable seeds both contours and pins the CI test suite.
LOG_LEVEL	variable	info	Shared log level for backend / gateway / validator / bot (debug|info|warn|error).
CADDY_SITE_ADDRESS	variable	:80	Caddy site address. Test: :80 (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME.
GM_BASICAUTH_USER	variable	gm	Username for the /_gm Basic-Auth.
GRAFANA_ROOT_URL	derived	/_gm/grafana/	Grafana root URL (sub-path serving). The deploy derives PUBLIC_BASE_URL + /_gm/grafana/; set the full https://<domain>/_gm/grafana/ only for a local run.
GRAFANA_ADMIN_PASSWORD	secret	admin	Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway.
TELEGRAM_GAME_CHANNEL_ID	variable	(empty)	The bot's game-channel id; empty/0 disables channel posts.
TELEGRAM_TEST_ENV	pinned	false	true routes the bot through Telegram's test environment (.../bot<token>/test/METHOD). The CI test contour pins this to true in ci.yaml (the contour is the test environment) — it is not a Gitea variable. Set it in .env for a local run; prod leaves it false.
TELEGRAM_API_BASE_URL	variable	(empty)	Override the Bot API host (a mock/self-hosted server); empty = https://api.telegram.org.
VITE_TELEGRAM_BOT_ID	variable	(empty)	UI build-arg: numeric bot id for the web Login Widget.
VITE_TELEGRAM_LINK	variable	(empty)	UI build-arg: friend-invite Mini App link (full URL, https://t.me/<bot>/<app> — <app> is the Mini App short name from BotFather).
VITE_TELEGRAM_GAME_CHANNEL_NAME	variable	(empty)	UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. https://t.me/Erudit_Game).
VITE_VK_APP_LINK	variable (shared)	(empty)	UI build-arg: the landing "Play on VK" link, the VK Mini App (full URL, https://vk.com/app<id>). One VK Mini App for all contours.
VITE_VK_APP_ID	variable (shared)	(empty)	VK ID "Web" app id (client_id) for VK web-login linking. Baked into the SPA authorize URL and reused as the gateway's GATEWAY_VK_ID_APP_ID. Empty disables the link.vk.* ops. One VK ID "Web" app for all contours.
VITE_VK_ID_REDIRECT_URL	derived	(empty)	VK ID trusted redirect URL — must match the app's registered redirect exactly. The deploy derives PUBLIC_BASE_URL + /app/ (used by the SPA and as the gateway's exchange redirect_uri); set it only for a local run.
GATEWAY_VK_APP_SECRET	secret (shared)	(empty)	The VK Mini App's protected key (client_secret): the gateway verifies the launch-parameter signature in-process under it (offline HMAC, no VK API call). Empty disables the VK auth path (auth.vk). One VK Mini App for all contours.
GATEWAY_VK_ID_CLIENT_SECRET	secret (shared)	(empty)	The VK ID "Web" app's protected key (client_secret) for the gateway's server-side confidential code exchange. A SEPARATE VK app from GATEWAY_VK_APP_SECRET (the Mini App). One VK ID "Web" app for all contours.
VITE_GATEWAY_URL	variable	(empty)	UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy).
SMTP_RELAY_HOST	variable (shared)	(empty)	Selectel SMTP relay host for confirm-code email. Empty leaves the backend on the log mailer (email disabled) — the contour still boots. One relay for every contour (limit 100 msgs / 5 min).
SMTP_RELAY_PORT	variable (shared)	465	Relay port. No client certificate is needed (the server cert is validated against the system roots).
SMTP_RELAY_TLS	variable (shared)	(empty)	Transport security: ssl (implicit TLS) or starttls. Empty derives it from the port (implicit on 465, STARTTLS otherwise); required for a relay on a non-standard port (e.g. Selectel's 1127 = SSL, 1126 = STARTTLS).
SMTP_RELAY_USER	secret (shared)	(empty)	Relay SMTP AUTH username.
SMTP_RELAY_PASS	secret (shared)	(empty)	Relay SMTP AUTH password.
SMTP_RELAY_FROM	variable	no-reply@localhost	Sender address. Must use the prod domain (no-reply@erudit-game.ru) — Selectel only accepts the verified sender domain — so it is the same on every contour.
PUBLIC_BASE_URL	variable	(empty)	Canonical public https origin for links in the email (the contour's own URL, e.g. https://erudit-game.ru). Required by the backend whenever SMTP_RELAY_HOST is set — it is never derived from a request Host header (anti-injection).
SMTP_RELAY_ADMIN_FROM	variable	(empty)	Backend operator-alert sender (new feedback / word complaints), distinct from the confirm-code From. Empty (with ADMIN_EMAIL) disables the alert worker.
ADMIN_EMAIL	variable	(empty)	Backend operator-alert recipient(s); several comma-separated addresses allowed.
SMTP_RELAY_SERVICE_FROM	variable	(empty)	Grafana infra-alert sender address.
SERVICE_EMAIL	variable	(empty)	Grafana infra-alert recipient(s); comma-separated allowed (read by the provisioned contact point via $__env{SERVICE_EMAIL}).
GRAFANA_SMTP_PORT	variable (shared)	(empty)	STARTTLS port Grafana dials on SMTP_RELAY_HOST (its client can't do the backend's implicit TLS), e.g. Selectel's 1126. Reuses SMTP_RELAY_HOST/USER/PASS.
GF_SMTP_ENABLED	variable	false	Set true to enable Grafana alert emails.

The six VITE_* are build-args baked into the gateway and landing images at build time (both targets share one UI build stage — keep the args identical so it is built once), so changing them requires a rebuild (--build), not just a restart.

Fixed internal wiring (not operator-set)

These are hard-wired in docker-compose.yml (no ${...}), pointing the services at each other on the internal network — listed here so they are not mistaken for missing config: BACKEND_POSTGRES_DSN (→ postgres, search_path=backend), GATEWAY_BACKEND_HTTP_URL/_GRPC_ADDR (→ backend), GATEWAY_VALIDATOR_ADDR (→ validator:9091), BACKEND_CONNECTOR_ADDR (→ the gateway bot-link relay gateway:9092), BACKEND_RENDERER_URL (→ renderer:8090, the image-render sidecar), the bot's TELEGRAM_GATEWAY_ADDR (→ gateway:9443, mTLS) with the GATEWAY_BOTLINK_* / TELEGRAM_BOTLINK_* cert paths under /certs (the mTLS material is generated by deploy/gen-certs.sh, gitignored, regenerated each deploy), and all services' *_OTEL_*_EXPORTER=otlp → OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317 (_INSECURE=true). The bot shares the VPN sidecar's netns: routing to the collector's / gateway's internal IP is fine (connected route), but its AWG_CONF must not set a DNS= directive — that hijacks resolv.conf and breaks resolving otelcol / gateway ("produced zero addresses"); without it the netns uses Docker's resolver, which resolves otelcol, gateway and api.telegram.org. GATEWAY_ADMIN_* is intentionally unset — caddy owns /_gm in the contour.

Bumping the dictionary version

The dictionary ships as a versioned release artifact (scrabble-dawg-vX.Y.Z.tar.gz) from scrabble-dictionary. The tag is a build-time input with no default in the images. It is a single Gitea repo variable — DICT_VERSION (unprefixed, shared across contours) — so a release bump is one edit:

• the CI test jobs read it via .gitea/workflows/ci.yaml's top-level env.DICT_VERSION: ${{ vars.DICT_VERSION }} (the unit/integration jobs download that dawg), and
• both deploy jobs feed the same variable to compose as the DICT_VERSION build-arg that bakes a fresh volume's seed.

For local builds set DICT_VERSION in deploy/.env (template: .env.example); a bare docker build needs --build-arg DICT_VERSION=vX.Y.Z. The Dockerfiles and compose carry no default — a missing value fails loudly instead of baking a stale tag.

Bumping the seed is a no-op on a live volume (the .seed_version marker wins — the seed-drift guard). A running contour/prod moves to a new release through the admin console /_gm/dictionary (upload the tarball, preview the per-variant diff, confirm); in-flight games keep their pinned version, new games use the new one (ARCHITECTURE.md §5).

Production rollout

Prod runs on two hosts (main = full stack + ACME on the domain; tg = the bot only, native Bot API, no VPN), one-time provisioned by ansible/ (docker, a non-sudo deploy user holding the CI key, key-only sshd, default-deny ufw, fail2ban). Re-run ansible/ after a host resize — it is idempotent.

To roll out: merge development → master (CI green), then run the prod-deploy workflow manually (Gitea → Actions → prod-deploy → run from master, input confirm=deploy). It builds + pushes the images to the registry, ships the compose/config/certs/env over SSH, deploys the main host with prod-deploy.sh (rolling, health-gated, auto-rollback to the previous tag; caddy is force-recreated on its roll so a bind-mounted Caddyfile change applies — its image is pinned and admin is off, so neither a new tag nor a hot reload would pick it up), then the bot host, then probes the public site. After master is green this workflow is the only thing that touches prod — nothing auto-deploys there. It runs four visible jobs: build → deploy-main → deploy-bot → verify (the per-service rolling shows in the deploy-main log).

Versioning. Each release is a git tag vX.Y.Z on master; the deploy stamps git describe --tags into every image tag, every binary (-ldflags → pkg/version → the service.version telemetry attribute) and the SPA About screen. Tag the release before running the deploy:

git tag -a v1.0.0 -m v1.0.0 && git push origin v1.0.0

Manual rollback (any time after a successful deploy). Run the prod-rollback workflow (Gitea → Actions → prod-rollback, confirm=rollback). Leave target_version blank to roll back to the previously deployed version (read from the host's PREVIOUS_TAG), or set it to a release tag from the Releases page. It re-deploys that already-published image rolling + health-gated — no rebuild, no DB migration (image rollback is DB-safe under the expand-contract rule). The registry keeps every release tag, so any prior release is reachable.

Migrations must be expand-contract (backward-compatible; goose is forward-only): the automatic rollback is image-only and never restores the DB. A deploy that changes backend/internal/postgres/migrations/ opens a maintenance window — the backend (sole writer) is stopped for a consistent pg_dump into /opt/scrabble/dumps before the new backend migrates. Manual DB restore (only if a migration was destructive): docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA backend CASCADE', then pipe the dump into the same psql, and redeploy the matching old tag.

bot-link cert rotation: regenerate (deploy/gen-certs.sh /tmp/c --force), reset the five PROD_BOTLINK_* secrets from /tmp/c, and re-run the workflow — both hosts redeploy together with the fresh CA.

Sizing / monitoring: the main host launches undersized (2 vCPU / 1.9 GiB); the prod overlay trims limits + GOMAXPROCS=2 + 7d Prometheus retention, and node_exporter feeds host memory to Grafana (/_gm/grafana/). Watch host memory and resize at Selectel when players arrive.

Shared Gitea set (one unprefixed entry each, used by every contour) — secrets: GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET, SMTP_RELAY_USER, SMTP_RELAY_PASS; variables: DICT_VERSION, SMTP_RELAY_HOST, SMTP_RELAY_PORT, SMTP_RELAY_TLS, GRAFANA_SMTP_PORT, VITE_VK_APP_LINK, VITE_VK_APP_ID. Derived from PUBLIC_BASE_URL at deploy (not stored): TELEGRAM_MINIAPP_URL, GRAFANA_ROOT_URL, VITE_VK_ID_REDIRECT_URL.

PROD_ Gitea set (per-contour, mirrors TEST_, mapped onto the unprefixed names above) — secrets: PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN, TELEGRAM_PROMO_BOT_TOKEN, EXPORT_SIGN_KEY, REGISTRY_PASSWORD, SSH_KEY, SSH_KNOWN_HOSTS, BOTLINK_CA, BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT, BOTLINK_BOT_KEY}; variables: PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER, LOG_LEVEL, TELEGRAM_GAME_CHANNEL_ID, TELEGRAM_CHAT_ID, TELEGRAM_SUPPORT_CHAT_ID, TELEGRAM_BOT_USERNAME, VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK, VITE_TELEGRAM_GAME_CHANNEL_NAME, SMTP_RELAY_FROM, PUBLIC_BASE_URL, SMTP_RELAY_ADMIN_FROM, ADMIN_EMAIL, SMTP_RELAY_SERVICE_FROM, SERVICE_EMAIL, GF_SMTP_ENABLED}. The test contour uses the same names under TEST_, minus the prod-only infra (MAIN_HOST/TG_HOST/REGISTRY_*/SSH_*/BOTLINK_*, which the test deploy generates or runs locally) and plus TEST_AWG_CONF (the bot's VPN egress).

Host-side setup (outside this repo)

• edge network must exist on the host (docker network create edge).
• Host caddy route <domain> → scrabble:80 (the in-compose caddy serves HTTP in the test contour; the host caddy terminates TLS). Not needed on prod, where the contour caddy owns TLS (set CADDY_SITE_ADDRESS to the domain).
• Branch protection requires the single status check CI / gate. The unit / integration / ui jobs are path-conditional (they skip when their code did not change), and the always-running gate job aggregates them (passing when each succeeded or was skipped), so a skipped job never blocks a merge. See ../CLAUDE.md "Branching & CI".