Split the contour across the two prod hosts and retune for the small main host: - Gate vpn+bot to the telegram-local profile. The CI test deploy now passes --profile telegram-local so the test contour still brings them; the prod main host omits both, and the prod bot runs standalone from docker-compose.bot.yml. - docker-compose.prod.yml (main-host overlay): publish caddy 80/443 (no host caddy in prod; caddy owns ACME) and gateway 9443 (the remote bot dials in over mTLS); GOMAXPROCS=2, smaller memory caps and 7d Prometheus retention for the 2 vCPU / 1.9 GiB host. It launches deliberately undersized; resize reactively. - docker-compose.bot.yml: standalone bot for the tg host (no VPN, OTLP off since otelcol is unreachable from there, dials the main host's bot-link). - Add node_exporter + a Prometheus scrape so host memory pressure (the OOM signal on the tight main host), not just per-container docker_stats, is visible. - Soften AWG_CONF to a default: only the profiled vpn sidecar consumes it, and compose interpolates profiled-out services too, so prod must not require it.
deploy
The full Scrabble contour: backend + gateway + the static landing + Postgres +
the Telegram validator + bot (the bot with a VPN sidecar) + the observability stack
(OTel Collector → Prometheus + Tempo → Grafana), fronted by a caddy that owns a single
/_gm Basic-Auth (the admin console + Grafana). Topology and the decision record are in
../docs/ARCHITECTURE.md §13; this file is the
operational reference for every environment variable.
Services
| Service | Image | Role |
|---|---|---|
caddy |
caddy:2-alpine |
Edge proxy (alias scrabble on edge): single /_gm Basic-Auth → admin console + Grafana; /app/, /telegram/ + the Connect path → gateway; the catch-all (incl. /) → landing. TLS per CADDY_SITE_ADDRESS. |
gateway |
built (gateway/Dockerfile, target gateway) |
Public edge; serves the embedded game SPA at /app/ + /telegram/; Connect-RPC edge. / redirects to /app/. |
landing |
built (gateway/Dockerfile, target landing) |
Static landing page at / (caddy:2-alpine + the shared Vite build, deploy/landing/Caddyfile); absorbs stray public paths. |
backend |
built (backend/Dockerfile) |
Domain service; bakes in the DAWG dictionaries; runs migrations at boot. |
postgres |
postgres:17-alpine |
Database (named volume, pg_isready healthcheck). |
validator |
built (platform/telegram/Dockerfile, target validator) |
Telegram HMAC validator (no VPN, no Bot API); internal gRPC at validator:9091. Game login depends only on this. |
vpn + bot |
sidecar + built (platform/telegram/Dockerfile, target bot) |
Telegram bot; egresses through the AmneziaWG sidecar; holds no inbound port — dials the gateway bot-link (mTLS) at gateway:9443. |
otelcol |
otel/opentelemetry-collector-contrib |
OTLP/gRPC :4317 → Prometheus scrape (:9464) + Tempo. |
prometheus |
prom/prometheus |
Metrics, 15d retention. |
tempo |
grafana/tempo |
Traces, 72h retention. |
grafana |
grafana/grafana |
Dashboards (provisioned), anonymous-admin behind caddy's /_gm/grafana. |
Networking: inter-service traffic is on the private internal network
(project-scoped DNS); only caddy joins the shared external edge network so the
host caddy can reach it at scrabble:80. edge must already exist on the host
(docker network create edge).
Run it
Locally — copy the template, fill the required values, bring it up:
cp deploy/.env.example deploy/.env # then edit deploy/.env
docker network create edge # once, if it does not exist
cd deploy && docker compose up -d --build
In CI (the test contour) — .gitea/workflows/ci.yaml's deploy job maps the
Gitea TEST_-prefixed secrets/variables onto the unprefixed names below and
runs docker compose up -d --build on the runner host. The prod deploy maps the
PROD_ set the same way. So a Gitea secret named TEST_POSTGRES_PASSWORD
feeds the compose's POSTGRES_PASSWORD, etc.
The deploy job also seeds the config files (caddy, otelcol, prometheus,
tempo, grafana) to a stable host path ($HOME/.scrabble-deploy) and sets
SCRABBLE_CONFIG_DIR to it before up. The runner's checkout is an ephemeral act
workspace that is removed after the job — binding config straight from it would
dangle the mounts in the long-lived containers (Grafana would log
no such file or directory). Locally SCRABBLE_CONFIG_DIR defaults to ., so the
compose binds from this directory.
Required variables
docker compose aborts immediately if any of these is unset (they use :?):
| Variable | Gitea kind | Purpose |
|---|---|---|
POSTGRES_PASSWORD |
secret | Postgres password (also embedded in BACKEND_POSTGRES_DSN). |
AWG_CONF |
secret | AmneziaWG config for the VPN sidecar (the bot's only Telegram egress in the test contour). Must not contain a DNS= line — it hijacks the shared netns's resolv.conf and breaks the bot resolving otelcol / gateway. Without it, Docker's resolver handles otelcol, gateway and api.telegram.org. |
GM_BASICAUTH_HASH |
secret | bcrypt hash gating /_gm (admin console + Grafana). Generate with docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'. |
TELEGRAM_MINIAPP_URL |
variable | The Mini App URL the bot hands out in deep links / buttons. |
Plus the bot token — TELEGRAM_BOT_TOKEN (secret), shared by the validator (HMAC
secret) and the bot (Bot API). It defaults to empty in compose, but both fail at
boot when it is empty.
Optional variables (with defaults)
| Variable | Gitea kind | Default | Purpose |
|---|---|---|---|
POSTGRES_DB |
variable | scrabble |
Database name. |
POSTGRES_USER |
variable | scrabble |
Database user. |
DICT_VERSION |
variable | v1.2.1 |
scrabble-dictionary release tag baked into the backend image as the seed for a fresh volume (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded .seed_version marker wins — the seed-drift guard, ARCHITECTURE.md §5). Set per contour as TEST_/PROD_DICT_VERSION. |
LOG_LEVEL |
variable | info |
Shared log level for backend / gateway / validator / bot (debug|info|warn|error). |
CADDY_SITE_ADDRESS |
variable | :80 |
Caddy site address. Test: :80 (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME. |
GM_BASICAUTH_USER |
variable | gm |
Username for the /_gm Basic-Auth. |
GRAFANA_ROOT_URL |
variable | /_gm/grafana/ |
Grafana root URL (sub-path serving). Set the full https://<domain>/_gm/grafana/ behind a real domain. |
GRAFANA_ADMIN_PASSWORD |
secret | admin |
Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway. |
TELEGRAM_GAME_CHANNEL_ID |
variable | (empty) | The bot's game-channel id; empty/0 disables channel posts. |
TELEGRAM_TEST_ENV |
pinned | false |
true routes the bot through Telegram's test environment (.../bot<token>/test/METHOD). The CI test contour pins this to true in ci.yaml (the contour is the test environment) — it is not a Gitea variable. Set it in .env for a local run; prod leaves it false. |
TELEGRAM_API_BASE_URL |
variable | (empty) | Override the Bot API host (a mock/self-hosted server); empty = https://api.telegram.org. |
VITE_TELEGRAM_BOT_ID |
variable | (empty) | UI build-arg: numeric bot id for the web Login Widget. |
VITE_TELEGRAM_LINK |
variable | (empty) | UI build-arg: friend-invite Mini App link (full URL, https://t.me/<bot>/<app> — <app> is the Mini App short name from BotFather). |
VITE_TELEGRAM_GAME_CHANNEL_NAME |
variable | (empty) | UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. https://t.me/Erudit_Game). |
VITE_GATEWAY_URL |
variable | (empty) | UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy). |
The five VITE_* are build-args baked into the gateway and landing images at
build time (both targets share one UI build stage — keep the args identical so it is
built once), so changing them requires a rebuild (--build), not just a restart.
Fixed internal wiring (not operator-set)
These are hard-wired in docker-compose.yml (no ${...}), pointing the services
at each other on the internal network — listed here so they are not mistaken for
missing config: BACKEND_POSTGRES_DSN (→ postgres, search_path=backend),
GATEWAY_BACKEND_HTTP_URL/_GRPC_ADDR (→ backend),
GATEWAY_VALIDATOR_ADDR (→ validator:9091), BACKEND_CONNECTOR_ADDR (→ the gateway
bot-link relay gateway:9092), the bot's TELEGRAM_GATEWAY_ADDR (→ gateway:9443,
mTLS) with the GATEWAY_BOTLINK_* / TELEGRAM_BOTLINK_* cert paths under /certs (the
mTLS material is generated by deploy/gen-certs.sh, gitignored, regenerated each
deploy), and all services' *_OTEL_*_EXPORTER=otlp →
OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317
(_INSECURE=true). The bot shares the VPN sidecar's netns: routing to the
collector's / gateway's internal IP is fine (connected route), but its AWG_CONF must
not set a DNS= directive — that hijacks resolv.conf and breaks resolving otelcol
/ gateway ("produced zero addresses"); without it the netns uses Docker's resolver,
which resolves otelcol, gateway and api.telegram.org. GATEWAY_ADMIN_* is
intentionally unset — caddy owns /_gm in the contour.
Host-side setup (outside this repo)
edgenetwork must exist on the host (docker network create edge).- Host caddy route
<domain> → scrabble:80(the in-compose caddy serves HTTP in the test contour; the host caddy terminates TLS). Not needed on prod, where the contour caddy owns TLS (setCADDY_SITE_ADDRESSto the domain). - Branch protection requires the single status check
CI / gate. Theunit/integration/uijobs are path-conditional (they skip when their code did not change), and the always-runninggatejob aggregates them (passing when each succeeded or was skipped), so a skipped job never blocks a merge. See../CLAUDE.md"Branching & CI".