feat(telegram): split connector into home validator + remote bot
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s

Move all Telegram egress off the main host. The single connector held the
bot token, long-polled Telegram and answered the gateway/backend over the
trusted internal network, so the whole component (including login validation)
shared fate with its VPN sidecar. Split it into two binaries that share the
token:

- cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only,
  never calls the Bot API. The gateway dials it for Telegram auth, so game
  login is now independent of Telegram reachability.
- cmd/bot (remote): Bot API long-poll + sendMessage, the only component
  reaching Telegram. It holds no inbound port — it dials the gateway over a
  new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send
  commands the gateway pushes.

The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget
(at-most-once, dropped if no bot is connected); the backend admin broadcasts
reach a gateway-served relay that forwards them and awaits the bot's ack
(SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one
inter-service link that leaves the trusted segment; validator<->gateway and
the relay stay plaintext internal. The bot is Telegram-rate-limited.

One bot now; the gateway bot registry, an owns_updates flag and per-command
ids leave seams for N later. Webhook rejected (one URL per token, adds inbound
+ a static address).

The unified test contour runs the split (the bot keeps its VPN sidecar and
dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh,
generated in CI). The prod wiring — the bot on a separate host (no VPN), the
gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH
deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX,
Stage 18).

Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway +
backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
This commit is contained in:
Ilia Denisov
2026-06-21 00:19:07 +02:00
parent 2a8717c930
commit 6aeb529f13
42 changed files with 3073 additions and 714 deletions
+81 -22
View File
@@ -1,6 +1,6 @@
# Full deploy descriptor for the Scrabble test contour: backend + gateway +
# Postgres + the Telegram connector (with its VPN sidecar) + the observability
# stack (OTel Collector -> Prometheus + Tempo -> Grafana). Driven by
# Postgres + the Telegram validator + bot (the bot with its VPN sidecar) + the
# observability stack (OTel Collector -> Prometheus + Tempo -> Grafana). Driven by
# .gitea/workflows/ci.yaml (`docker compose up -d --build`); env values are
# interpolated from Gitea Actions TEST_ secrets/variables exported by the deploy
# job (see deploy/.env.example for the unprefixed names).
@@ -19,8 +19,10 @@
# the test contour; the host caddy terminates TLS and forwards. For prod
# (no host caddy) set CADDY_SITE_ADDRESS to the domain so the caddy
# does its own ACME — the contour is then self-contained.
# - The connector egresses to api.telegram.org through the `vpn` sidecar
# (network_mode: service:vpn); it answers internal gRPC at `telegram:9091`.
# - The validator answers internal gRPC at `validator:9091` (no VPN, HMAC only).
# The bot egresses to api.telegram.org through the `vpn` sidecar (network_mode:
# service:vpn) and dials the gateway bot-link (mTLS) at `gateway:9443`. The
# backend admin relay reaches the gateway at `gateway:9092` (plaintext).
name: scrabble
# Bound every container's json-file logs. R7 measured the backend emitting a
@@ -82,7 +84,9 @@ services:
BACKEND_POSTGRES_MAX_OPEN_CONNS: "40"
BACKEND_HTTP_ADDR: ":8080"
BACKEND_GRPC_ADDR: ":9090"
BACKEND_CONNECTOR_ADDR: telegram:9091
# Admin broadcasts go to the gateway's bot-link relay, which forwards them to
# the remote bot and reports back whether they were delivered.
BACKEND_CONNECTOR_ADDR: gateway:9092
BACKEND_LOG_LEVEL: ${LOG_LEVEL:-info}
BACKEND_SERVICE_NAME: scrabble-backend
BACKEND_OTEL_TRACES_EXPORTER: otlp
@@ -135,7 +139,17 @@ services:
GATEWAY_HTTP_ADDR: ":8081"
GATEWAY_BACKEND_HTTP_URL: http://backend:8080
GATEWAY_BACKEND_GRPC_ADDR: backend:9090
GATEWAY_CONNECTOR_ADDR: telegram:9091
# Telegram auth validates against the home validator (plaintext, internal).
GATEWAY_VALIDATOR_ADDR: validator:9091
# The reverse bot-link: the bot dials :9443 over mTLS; the backend admin relay
# reaches the gateway at :9092 (plaintext, internal). In the test contour both
# listeners stay on the internal network (the bot shares the VPN netns); in prod
# the bot is a separate host and :9443 is published with public certificates.
GATEWAY_BOTLINK_ADDR: ":9443"
GATEWAY_BOTLINK_RELAY_ADDR: ":9092"
GATEWAY_BOTLINK_TLS_CERT: /certs/gateway.crt
GATEWAY_BOTLINK_TLS_KEY: /certs/gateway.key
GATEWAY_BOTLINK_TLS_CA: /certs/ca.crt
GATEWAY_LOG_LEVEL: ${LOG_LEVEL:-info}
GATEWAY_SERVICE_NAME: scrabble-gateway
GATEWAY_OTEL_TRACES_EXPORTER: otlp
@@ -146,6 +160,10 @@ services:
GOMAXPROCS: "3"
# GATEWAY_ADMIN_* intentionally unset: in the deployed contour the front
# caddy owns the /_gm Basic-Auth and routes /_gm to the backend directly.
# The bot-link mTLS material (CA + gateway server leaf). Generated by
# deploy/gen-certs.sh for the test contour; supplied from PROD_ secrets in prod.
volumes:
- ${SCRABBLE_CONFIG_DIR:-.}/certs:/certs:ro
# R7 tuned: the gateway holds one h2c connection per player, so at 500 players it
# bursts into a 2-core cap (~2.49% transport_error on game.state); 3 cores absorbs
# the bursts. Per-connection overhead is the realistic prod cost — size for it.
@@ -182,7 +200,39 @@ services:
memory: 128M
networks: [internal]
# --- Telegram connector (egress via the VPN sidecar) -----------------------
# --- Telegram validator (home; HMAC only, no VPN, no Telegram egress) -------
# The validator holds the bot token solely as the HMAC secret and never reaches
# the Bot API, so it runs on the main network with no VPN. Game login validates
# against it and stays up even when the remote bot or the bot-link is down.
validator:
container_name: scrabble-telegram-validator
image: scrabble-telegram-validator:latest
build:
context: ..
dockerfile: platform/telegram/Dockerfile
target: validator
restart: unless-stopped
logging: *default-logging
environment:
# The token is the HMAC secret only; the validator never calls the Bot API. An
# empty value crash-loops only the validator; the rest of the contour comes up.
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
TELEGRAM_VALIDATOR_GRPC_ADDR: ":9091"
TELEGRAM_LOG_LEVEL: ${LOG_LEVEL:-info}
TELEGRAM_SERVICE_NAME: scrabble-telegram-validator
TELEGRAM_OTEL_TRACES_EXPORTER: otlp
TELEGRAM_OTEL_METRICS_EXPORTER: otlp
OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4317
OTEL_EXPORTER_OTLP_INSECURE: "true"
GOMAXPROCS: "1"
deploy:
resources:
limits:
cpus: "1.0"
memory: 128M
networks: [internal]
# --- Telegram bot (egress via the VPN sidecar in test; dials the gateway) ---
vpn:
container_name: scrabble-telegram-vpn
image: docker.iliadenisov.ru/developer/amneziawg-sidecar:latest
@@ -195,40 +245,49 @@ services:
internal:
aliases: [telegram]
telegram:
container_name: scrabble-telegram
image: scrabble-telegram:latest
bot:
container_name: scrabble-telegram-bot
image: scrabble-telegram-bot:latest
build:
context: ..
dockerfile: platform/telegram/Dockerfile
target: bot
restart: unless-stopped
logging: *default-logging
depends_on: [vpn]
network_mode: "service:vpn"
environment:
# The bot token lives ONLY in this container (ARCHITECTURE.md §12). The connector
# requires it at boot; an empty value leaves the Telegram side down while the rest
# of the contour still comes up.
# The bot token lives on the bot host (ARCHITECTURE.md §12). The bot requires it
# at boot; an empty value leaves the bot down while the rest of the contour comes up.
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
TELEGRAM_GAME_CHANNEL_ID: ${TELEGRAM_GAME_CHANNEL_ID:-}
TELEGRAM_MINIAPP_URL: ${TELEGRAM_MINIAPP_URL:?set TELEGRAM_MINIAPP_URL}
TELEGRAM_GRPC_ADDR: ":9091"
TELEGRAM_TEST_ENV: ${TELEGRAM_TEST_ENV:-false}
TELEGRAM_API_BASE_URL: ${TELEGRAM_API_BASE_URL:-}
TELEGRAM_OWNS_UPDATES: "true"
# The bot dials the gateway bot-link over mTLS. In test it reaches the gateway by
# its internal service name through the VPN netns (Docker resolver, off-tunnel,
# like otelcol); in prod it is a separate host dialing the gateway's public port.
TELEGRAM_GATEWAY_ADDR: gateway:9443
TELEGRAM_BOTLINK_SERVER_NAME: gateway
TELEGRAM_BOTLINK_TLS_CERT: /certs/bot.crt
TELEGRAM_BOTLINK_TLS_KEY: /certs/bot.key
TELEGRAM_BOTLINK_TLS_CA: /certs/ca.crt
TELEGRAM_LOG_LEVEL: ${LOG_LEVEL:-info}
TELEGRAM_SERVICE_NAME: scrabble-telegram
# The connector shares the VPN sidecar's netns. Routing to the collector's
# internal IP stays off the tunnel (connected route), but the sidecar's DNS
# hijacks name resolution: AWG_CONF must NOT carry a `DNS=` directive, else
# `otelcol` won't resolve ("produced zero addresses"). Without DNS= the netns
# uses Docker's resolver, which resolves both otelcol and api.telegram.org
# (see deploy/README.md).
TELEGRAM_SERVICE_NAME: scrabble-telegram-bot
# The bot shares the VPN sidecar's netns. Routing to internal IPs stays off the
# tunnel (connected route), but the sidecar's DNS hijacks name resolution:
# AWG_CONF must NOT carry a `DNS=` directive, else `otelcol` and `gateway` won't
# resolve. Without DNS= the netns uses Docker's resolver, which resolves the
# internal services and api.telegram.org (see deploy/README.md).
TELEGRAM_OTEL_TRACES_EXPORTER: otlp
TELEGRAM_OTEL_METRICS_EXPORTER: otlp
OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4317
OTEL_EXPORTER_OTLP_INSECURE: "true"
# The connector is light (the stress run does not drive Telegram); one P suffices.
# The bot is light (the stress run does not drive Telegram); one P suffices.
GOMAXPROCS: "1"
volumes:
- ${SCRABBLE_CONFIG_DIR:-.}/certs:/certs:ro
deploy:
resources:
limits: