feat(telegram): split connector into home validator + remote bot
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s

Move all Telegram egress off the main host. The single connector held the
bot token, long-polled Telegram and answered the gateway/backend over the
trusted internal network, so the whole component (including login validation)
shared fate with its VPN sidecar. Split it into two binaries that share the
token:

- cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only,
  never calls the Bot API. The gateway dials it for Telegram auth, so game
  login is now independent of Telegram reachability.
- cmd/bot (remote): Bot API long-poll + sendMessage, the only component
  reaching Telegram. It holds no inbound port — it dials the gateway over a
  new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send
  commands the gateway pushes.

The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget
(at-most-once, dropped if no bot is connected); the backend admin broadcasts
reach a gateway-served relay that forwards them and awaits the bot's ack
(SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one
inter-service link that leaves the trusted segment; validator<->gateway and
the relay stay plaintext internal. The bot is Telegram-rate-limited.

One bot now; the gateway bot registry, an owns_updates flag and per-command
ids leave seams for N later. Webhook rejected (one URL per token, adds inbound
+ a static address).

The unified test contour runs the split (the bot keeps its VPN sidecar and
dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh,
generated in CI). The prod wiring — the bot on a separate host (no VPN), the
gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH
deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX,
Stage 18).

Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway +
backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
This commit is contained in:
Ilia Denisov
2026-06-21 00:19:07 +02:00
parent 2a8717c930
commit 6aeb529f13
42 changed files with 3073 additions and 714 deletions
+22 -16
View File
@@ -1,8 +1,8 @@
# deploy
The full Scrabble contour: `backend` + `gateway` + the static `landing` + Postgres +
the Telegram connector (with a VPN sidecar) + the observability stack (OTel
Collector → Prometheus + Tempo → Grafana), fronted by a **caddy** that owns a single
the Telegram `validator` + `bot` (the bot with a VPN sidecar) + the observability stack
(OTel Collector → Prometheus + Tempo → Grafana), fronted by a **caddy** that owns a single
`/_gm` Basic-Auth (the admin console + Grafana). Topology and the decision record are in
[`../docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) §13; this file is the
operational reference for **every environment variable**.
@@ -16,7 +16,8 @@ operational reference for **every environment variable**.
| `landing` | built (`gateway/Dockerfile`, target `landing`) | Static landing page at `/` (caddy:2-alpine + the shared Vite build, `deploy/landing/Caddyfile`); absorbs stray public paths. |
| `backend` | built (`backend/Dockerfile`) | Domain service; bakes in the DAWG dictionaries; runs migrations at boot. |
| `postgres` | `postgres:17-alpine` | Database (named volume, `pg_isready` healthcheck). |
| `vpn` + `telegram` | sidecar + built (`platform/telegram/Dockerfile`) | Telegram connector; egresses through the AmneziaWG sidecar; internal gRPC at `telegram:9091`. |
| `validator` | built (`platform/telegram/Dockerfile`, target `validator`) | Telegram HMAC validator (no VPN, no Bot API); internal gRPC at `validator:9091`. Game login depends only on this. |
| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot; egresses through the AmneziaWG sidecar; holds no inbound port — dials the gateway bot-link (mTLS) at `gateway:9443`. |
| `otelcol` | `otel/opentelemetry-collector-contrib` | OTLP/gRPC `:4317` → Prometheus scrape (`:9464`) + Tempo. |
| `prometheus` | `prom/prometheus` | Metrics, 15d retention. |
| `tempo` | `grafana/tempo` | Traces, 72h retention. |
@@ -58,12 +59,13 @@ compose binds from this directory.
| Variable | Gitea kind | Purpose |
| --- | --- | --- |
| `POSTGRES_PASSWORD` | secret | Postgres password (also embedded in `BACKEND_POSTGRES_DSN`). |
| `AWG_CONF` | secret | AmneziaWG config for the VPN sidecar (the connector's only egress). **Must not contain a `DNS=` line** — it hijacks the shared netns's resolv.conf and breaks the connector resolving `otelcol` (telemetry export). Without it, Docker's resolver handles both `otelcol` and `api.telegram.org`. |
| `AWG_CONF` | secret | AmneziaWG config for the VPN sidecar (the bot's only Telegram egress in the test contour). **Must not contain a `DNS=` line** — it hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`. Without it, Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`. |
| `GM_BASICAUTH_HASH` | secret | bcrypt hash gating `/_gm` (admin console + Grafana). Generate with `docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'`. |
| `TELEGRAM_MINIAPP_URL` | variable | The Mini App URL the connector hands out in deep links / buttons. |
| `TELEGRAM_MINIAPP_URL` | variable | The Mini App URL the bot hands out in deep links / buttons. |
**Plus the bot token**`TELEGRAM_BOT_TOKEN` (secret). It defaults to empty in
compose, but the connector **fails at boot** when it is empty.
**Plus the bot token**`TELEGRAM_BOT_TOKEN` (secret), shared by the validator (HMAC
secret) and the bot (Bot API). It defaults to empty in compose, but both **fail at
boot** when it is empty.
## Optional variables (with defaults)
@@ -72,7 +74,7 @@ compose, but the connector **fails at boot** when it is empty.
| `POSTGRES_DB` | variable | `scrabble` | Database name. |
| `POSTGRES_USER` | variable | `scrabble` | Database user. |
| `DICT_VERSION` | variable | `v1.2.1` | `scrabble-dictionary` release tag baked into the backend image as the **seed for a fresh volume** (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded `.seed_version` marker wins — the seed-drift guard, ARCHITECTURE.md §5). Set per contour as `TEST_`/`PROD_DICT_VERSION`. |
| `LOG_LEVEL` | variable | `info` | Shared log level for backend / gateway / connector (`debug\|info\|warn\|error`). |
| `LOG_LEVEL` | variable | `info` | Shared log level for backend / gateway / validator / bot (`debug\|info\|warn\|error`). |
| `CADDY_SITE_ADDRESS` | variable | `:80` | Caddy site address. Test: `:80` (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME. |
| `GM_BASICAUTH_USER` | variable | `gm` | Username for the `/_gm` Basic-Auth. |
| `GRAFANA_ROOT_URL` | variable | `/_gm/grafana/` | Grafana root URL (sub-path serving). Set the full `https://<domain>/_gm/grafana/` behind a real domain. |
@@ -95,14 +97,18 @@ These are hard-wired in `docker-compose.yml` (no `${...}`), pointing the service
at each other on the `internal` network — listed here so they are not mistaken for
missing config: `BACKEND_POSTGRES_DSN` (→ `postgres`, `search_path=backend`),
`GATEWAY_BACKEND_HTTP_URL`/`_GRPC_ADDR` (→ `backend`),
`GATEWAY_CONNECTOR_ADDR`/`BACKEND_CONNECTOR_ADDR` (→ `telegram:9091`), and all three
services' `*_OTEL_*_EXPORTER=otlp` `OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317`
(`_INSECURE=true`). The connector shares the VPN sidecar's netns: routing to the
collector's internal IP is fine (connected route), but its `AWG_CONF` must **not**
set a `DNS=` directive — that hijacks resolv.conf and breaks resolving `otelcol`
("produced zero addresses"); without it the netns uses Docker's resolver, which
resolves both `otelcol` and `api.telegram.org`. `GATEWAY_ADMIN_*` is intentionally
**unset** — caddy owns `/_gm` in the contour.
`GATEWAY_VALIDATOR_ADDR` (→ `validator:9091`), `BACKEND_CONNECTOR_ADDR` (→ the gateway
bot-link relay `gateway:9092`), the bot's `TELEGRAM_GATEWAY_ADDR` (→ `gateway:9443`,
mTLS) with the `GATEWAY_BOTLINK_*` / `TELEGRAM_BOTLINK_*` cert paths under `/certs` (the
mTLS material is generated by `deploy/gen-certs.sh`, gitignored, regenerated each
deploy), and all services' `*_OTEL_*_EXPORTER=otlp`
`OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317`
(`_INSECURE=true`). The bot shares the VPN sidecar's netns: routing to the
collector's / gateway's internal IP is fine (connected route), but its `AWG_CONF` must
**not** set a `DNS=` directive — that hijacks resolv.conf and breaks resolving `otelcol`
/ `gateway` ("produced zero addresses"); without it the netns uses Docker's resolver,
which resolves `otelcol`, `gateway` and `api.telegram.org`. `GATEWAY_ADMIN_*` is
intentionally **unset** — caddy owns `/_gm` in the contour.
## Host-side setup (outside this repo)