Files
scrabble-game/deploy
Ilia Denisov 6aeb529f13
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s
feat(telegram): split connector into home validator + remote bot
Move all Telegram egress off the main host. The single connector held the
bot token, long-polled Telegram and answered the gateway/backend over the
trusted internal network, so the whole component (including login validation)
shared fate with its VPN sidecar. Split it into two binaries that share the
token:

- cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only,
  never calls the Bot API. The gateway dials it for Telegram auth, so game
  login is now independent of Telegram reachability.
- cmd/bot (remote): Bot API long-poll + sendMessage, the only component
  reaching Telegram. It holds no inbound port — it dials the gateway over a
  new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send
  commands the gateway pushes.

The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget
(at-most-once, dropped if no bot is connected); the backend admin broadcasts
reach a gateway-served relay that forwards them and awaits the bot's ack
(SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one
inter-service link that leaves the trusted segment; validator<->gateway and
the relay stay plaintext internal. The bot is Telegram-rate-limited.

One bot now; the gateway bot registry, an owns_updates flag and per-command
ids leave seams for N later. Webhook rejected (one URL per token, adds inbound
+ a static address).

The unified test contour runs the split (the bot keeps its VPN sidecar and
dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh,
generated in CI). The prod wiring — the bot on a separate host (no VPN), the
gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH
deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX,
Stage 18).

Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway +
backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
2026-06-21 00:19:07 +02:00
..

deploy

The full Scrabble contour: backend + gateway + the static landing + Postgres + the Telegram validator + bot (the bot with a VPN sidecar) + the observability stack (OTel Collector → Prometheus + Tempo → Grafana), fronted by a caddy that owns a single /_gm Basic-Auth (the admin console + Grafana). Topology and the decision record are in ../docs/ARCHITECTURE.md §13; this file is the operational reference for every environment variable.

Services

Service Image Role
caddy caddy:2-alpine Edge proxy (alias scrabble on edge): single /_gm Basic-Auth → admin console + Grafana; /app/, /telegram/ + the Connect path → gateway; the catch-all (incl. /) → landing. TLS per CADDY_SITE_ADDRESS.
gateway built (gateway/Dockerfile, target gateway) Public edge; serves the embedded game SPA at /app/ + /telegram/; Connect-RPC edge. / redirects to /app/.
landing built (gateway/Dockerfile, target landing) Static landing page at / (caddy:2-alpine + the shared Vite build, deploy/landing/Caddyfile); absorbs stray public paths.
backend built (backend/Dockerfile) Domain service; bakes in the DAWG dictionaries; runs migrations at boot.
postgres postgres:17-alpine Database (named volume, pg_isready healthcheck).
validator built (platform/telegram/Dockerfile, target validator) Telegram HMAC validator (no VPN, no Bot API); internal gRPC at validator:9091. Game login depends only on this.
vpn + bot sidecar + built (platform/telegram/Dockerfile, target bot) Telegram bot; egresses through the AmneziaWG sidecar; holds no inbound port — dials the gateway bot-link (mTLS) at gateway:9443.
otelcol otel/opentelemetry-collector-contrib OTLP/gRPC :4317 → Prometheus scrape (:9464) + Tempo.
prometheus prom/prometheus Metrics, 15d retention.
tempo grafana/tempo Traces, 72h retention.
grafana grafana/grafana Dashboards (provisioned), anonymous-admin behind caddy's /_gm/grafana.

Networking: inter-service traffic is on the private internal network (project-scoped DNS); only caddy joins the shared external edge network so the host caddy can reach it at scrabble:80. edge must already exist on the host (docker network create edge).

Run it

Locally — copy the template, fill the required values, bring it up:

cp deploy/.env.example deploy/.env     # then edit deploy/.env
docker network create edge             # once, if it does not exist
cd deploy && docker compose up -d --build

In CI (the test contour) — .gitea/workflows/ci.yaml's deploy job maps the Gitea TEST_-prefixed secrets/variables onto the unprefixed names below and runs docker compose up -d --build on the runner host. The prod deploy maps the PROD_ set the same way. So a Gitea secret named TEST_POSTGRES_PASSWORD feeds the compose's POSTGRES_PASSWORD, etc.

The deploy job also seeds the config files (caddy, otelcol, prometheus, tempo, grafana) to a stable host path ($HOME/.scrabble-deploy) and sets SCRABBLE_CONFIG_DIR to it before up. The runner's checkout is an ephemeral act workspace that is removed after the job — binding config straight from it would dangle the mounts in the long-lived containers (Grafana would log no such file or directory). Locally SCRABBLE_CONFIG_DIR defaults to ., so the compose binds from this directory.

Required variables

docker compose aborts immediately if any of these is unset (they use :?):

Variable Gitea kind Purpose
POSTGRES_PASSWORD secret Postgres password (also embedded in BACKEND_POSTGRES_DSN).
AWG_CONF secret AmneziaWG config for the VPN sidecar (the bot's only Telegram egress in the test contour). Must not contain a DNS= line — it hijacks the shared netns's resolv.conf and breaks the bot resolving otelcol / gateway. Without it, Docker's resolver handles otelcol, gateway and api.telegram.org.
GM_BASICAUTH_HASH secret bcrypt hash gating /_gm (admin console + Grafana). Generate with docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'.
TELEGRAM_MINIAPP_URL variable The Mini App URL the bot hands out in deep links / buttons.

Plus the bot tokenTELEGRAM_BOT_TOKEN (secret), shared by the validator (HMAC secret) and the bot (Bot API). It defaults to empty in compose, but both fail at boot when it is empty.

Optional variables (with defaults)

Variable Gitea kind Default Purpose
POSTGRES_DB variable scrabble Database name.
POSTGRES_USER variable scrabble Database user.
DICT_VERSION variable v1.2.1 scrabble-dictionary release tag baked into the backend image as the seed for a fresh volume (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded .seed_version marker wins — the seed-drift guard, ARCHITECTURE.md §5). Set per contour as TEST_/PROD_DICT_VERSION.
LOG_LEVEL variable info Shared log level for backend / gateway / validator / bot (debug|info|warn|error).
CADDY_SITE_ADDRESS variable :80 Caddy site address. Test: :80 (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME.
GM_BASICAUTH_USER variable gm Username for the /_gm Basic-Auth.
GRAFANA_ROOT_URL variable /_gm/grafana/ Grafana root URL (sub-path serving). Set the full https://<domain>/_gm/grafana/ behind a real domain.
GRAFANA_ADMIN_PASSWORD secret admin Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway.
TELEGRAM_GAME_CHANNEL_ID variable (empty) The bot's game-channel id; empty/0 disables channel posts.
TELEGRAM_TEST_ENV pinned false true routes the bot through Telegram's test environment (.../bot<token>/test/METHOD). The CI test contour pins this to true in ci.yaml (the contour is the test environment) — it is not a Gitea variable. Set it in .env for a local run; prod leaves it false.
TELEGRAM_API_BASE_URL variable (empty) Override the Bot API host (a mock/self-hosted server); empty = https://api.telegram.org.
VITE_TELEGRAM_BOT_ID variable (empty) UI build-arg: numeric bot id for the web Login Widget.
VITE_TELEGRAM_LINK variable (empty) UI build-arg: friend-invite Mini App link (full URL, https://t.me/<bot>/<app><app> is the Mini App short name from BotFather).
VITE_TELEGRAM_GAME_CHANNEL_NAME variable (empty) UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. https://t.me/Erudit_Game).
VITE_GATEWAY_URL variable (empty) UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy).

The five VITE_* are build-args baked into the gateway and landing images at build time (both targets share one UI build stage — keep the args identical so it is built once), so changing them requires a rebuild (--build), not just a restart.

Fixed internal wiring (not operator-set)

These are hard-wired in docker-compose.yml (no ${...}), pointing the services at each other on the internal network — listed here so they are not mistaken for missing config: BACKEND_POSTGRES_DSN (→ postgres, search_path=backend), GATEWAY_BACKEND_HTTP_URL/_GRPC_ADDR (→ backend), GATEWAY_VALIDATOR_ADDR (→ validator:9091), BACKEND_CONNECTOR_ADDR (→ the gateway bot-link relay gateway:9092), the bot's TELEGRAM_GATEWAY_ADDR (→ gateway:9443, mTLS) with the GATEWAY_BOTLINK_* / TELEGRAM_BOTLINK_* cert paths under /certs (the mTLS material is generated by deploy/gen-certs.sh, gitignored, regenerated each deploy), and all services' *_OTEL_*_EXPORTER=otlpOTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317 (_INSECURE=true). The bot shares the VPN sidecar's netns: routing to the collector's / gateway's internal IP is fine (connected route), but its AWG_CONF must not set a DNS= directive — that hijacks resolv.conf and breaks resolving otelcol / gateway ("produced zero addresses"); without it the netns uses Docker's resolver, which resolves otelcol, gateway and api.telegram.org. GATEWAY_ADMIN_* is intentionally unset — caddy owns /_gm in the contour.

Host-side setup (outside this repo)

  • edge network must exist on the host (docker network create edge).
  • Host caddy route <domain> → scrabble:80 (the in-compose caddy serves HTTP in the test contour; the host caddy terminates TLS). Not needed on prod, where the contour caddy owns TLS (set CADDY_SITE_ADDRESS to the domain).
  • Branch protection requires the single status check CI / gate. The unit / integration / ui jobs are path-conditional (they skip when their code did not change), and the always-running gate job aggregates them (passing when each succeeded or was skipped), so a skipped job never blocks a merge. See ../CLAUDE.md "Branching & CI".