feat(telegram): split connector into home validator + remote bot
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s
Move all Telegram egress off the main host. The single connector held the bot token, long-polled Telegram and answered the gateway/backend over the trusted internal network, so the whole component (including login validation) shared fate with its VPN sidecar. Split it into two binaries that share the token: - cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only, never calls the Bot API. The gateway dials it for Telegram auth, so game login is now independent of Telegram reachability. - cmd/bot (remote): Bot API long-poll + sendMessage, the only component reaching Telegram. It holds no inbound port — it dials the gateway over a new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send commands the gateway pushes. The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget (at-most-once, dropped if no bot is connected); the backend admin broadcasts reach a gateway-served relay that forwards them and awaits the bot's ack (SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one inter-service link that leaves the trusted segment; validator<->gateway and the relay stay plaintext internal. The bot is Telegram-rate-limited. One bot now; the gateway bot registry, an owns_updates flag and per-command ids leave seams for N later. Webhook rejected (one URL per token, adds inbound + a static address). The unified test contour runs the split (the bot keeps its VPN sidecar and dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh, generated in CI). The prod wiring — the bot on a separate host (no VPN), the gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX, Stage 18). Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway + backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
This commit is contained in:
+81
-22
@@ -1,6 +1,6 @@
|
||||
# Full deploy descriptor for the Scrabble test contour: backend + gateway +
|
||||
# Postgres + the Telegram connector (with its VPN sidecar) + the observability
|
||||
# stack (OTel Collector -> Prometheus + Tempo -> Grafana). Driven by
|
||||
# Postgres + the Telegram validator + bot (the bot with its VPN sidecar) + the
|
||||
# observability stack (OTel Collector -> Prometheus + Tempo -> Grafana). Driven by
|
||||
# .gitea/workflows/ci.yaml (`docker compose up -d --build`); env values are
|
||||
# interpolated from Gitea Actions TEST_ secrets/variables exported by the deploy
|
||||
# job (see deploy/.env.example for the unprefixed names).
|
||||
@@ -19,8 +19,10 @@
|
||||
# the test contour; the host caddy terminates TLS and forwards. For prod
|
||||
# (no host caddy) set CADDY_SITE_ADDRESS to the domain so the caddy
|
||||
# does its own ACME — the contour is then self-contained.
|
||||
# - The connector egresses to api.telegram.org through the `vpn` sidecar
|
||||
# (network_mode: service:vpn); it answers internal gRPC at `telegram:9091`.
|
||||
# - The validator answers internal gRPC at `validator:9091` (no VPN, HMAC only).
|
||||
# The bot egresses to api.telegram.org through the `vpn` sidecar (network_mode:
|
||||
# service:vpn) and dials the gateway bot-link (mTLS) at `gateway:9443`. The
|
||||
# backend admin relay reaches the gateway at `gateway:9092` (plaintext).
|
||||
name: scrabble
|
||||
|
||||
# Bound every container's json-file logs. R7 measured the backend emitting a
|
||||
@@ -82,7 +84,9 @@ services:
|
||||
BACKEND_POSTGRES_MAX_OPEN_CONNS: "40"
|
||||
BACKEND_HTTP_ADDR: ":8080"
|
||||
BACKEND_GRPC_ADDR: ":9090"
|
||||
BACKEND_CONNECTOR_ADDR: telegram:9091
|
||||
# Admin broadcasts go to the gateway's bot-link relay, which forwards them to
|
||||
# the remote bot and reports back whether they were delivered.
|
||||
BACKEND_CONNECTOR_ADDR: gateway:9092
|
||||
BACKEND_LOG_LEVEL: ${LOG_LEVEL:-info}
|
||||
BACKEND_SERVICE_NAME: scrabble-backend
|
||||
BACKEND_OTEL_TRACES_EXPORTER: otlp
|
||||
@@ -135,7 +139,17 @@ services:
|
||||
GATEWAY_HTTP_ADDR: ":8081"
|
||||
GATEWAY_BACKEND_HTTP_URL: http://backend:8080
|
||||
GATEWAY_BACKEND_GRPC_ADDR: backend:9090
|
||||
GATEWAY_CONNECTOR_ADDR: telegram:9091
|
||||
# Telegram auth validates against the home validator (plaintext, internal).
|
||||
GATEWAY_VALIDATOR_ADDR: validator:9091
|
||||
# The reverse bot-link: the bot dials :9443 over mTLS; the backend admin relay
|
||||
# reaches the gateway at :9092 (plaintext, internal). In the test contour both
|
||||
# listeners stay on the internal network (the bot shares the VPN netns); in prod
|
||||
# the bot is a separate host and :9443 is published with public certificates.
|
||||
GATEWAY_BOTLINK_ADDR: ":9443"
|
||||
GATEWAY_BOTLINK_RELAY_ADDR: ":9092"
|
||||
GATEWAY_BOTLINK_TLS_CERT: /certs/gateway.crt
|
||||
GATEWAY_BOTLINK_TLS_KEY: /certs/gateway.key
|
||||
GATEWAY_BOTLINK_TLS_CA: /certs/ca.crt
|
||||
GATEWAY_LOG_LEVEL: ${LOG_LEVEL:-info}
|
||||
GATEWAY_SERVICE_NAME: scrabble-gateway
|
||||
GATEWAY_OTEL_TRACES_EXPORTER: otlp
|
||||
@@ -146,6 +160,10 @@ services:
|
||||
GOMAXPROCS: "3"
|
||||
# GATEWAY_ADMIN_* intentionally unset: in the deployed contour the front
|
||||
# caddy owns the /_gm Basic-Auth and routes /_gm to the backend directly.
|
||||
# The bot-link mTLS material (CA + gateway server leaf). Generated by
|
||||
# deploy/gen-certs.sh for the test contour; supplied from PROD_ secrets in prod.
|
||||
volumes:
|
||||
- ${SCRABBLE_CONFIG_DIR:-.}/certs:/certs:ro
|
||||
# R7 tuned: the gateway holds one h2c connection per player, so at 500 players it
|
||||
# bursts into a 2-core cap (~2.49% transport_error on game.state); 3 cores absorbs
|
||||
# the bursts. Per-connection overhead is the realistic prod cost — size for it.
|
||||
@@ -182,7 +200,39 @@ services:
|
||||
memory: 128M
|
||||
networks: [internal]
|
||||
|
||||
# --- Telegram connector (egress via the VPN sidecar) -----------------------
|
||||
# --- Telegram validator (home; HMAC only, no VPN, no Telegram egress) -------
|
||||
# The validator holds the bot token solely as the HMAC secret and never reaches
|
||||
# the Bot API, so it runs on the main network with no VPN. Game login validates
|
||||
# against it and stays up even when the remote bot or the bot-link is down.
|
||||
validator:
|
||||
container_name: scrabble-telegram-validator
|
||||
image: scrabble-telegram-validator:latest
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: platform/telegram/Dockerfile
|
||||
target: validator
|
||||
restart: unless-stopped
|
||||
logging: *default-logging
|
||||
environment:
|
||||
# The token is the HMAC secret only; the validator never calls the Bot API. An
|
||||
# empty value crash-loops only the validator; the rest of the contour comes up.
|
||||
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
|
||||
TELEGRAM_VALIDATOR_GRPC_ADDR: ":9091"
|
||||
TELEGRAM_LOG_LEVEL: ${LOG_LEVEL:-info}
|
||||
TELEGRAM_SERVICE_NAME: scrabble-telegram-validator
|
||||
TELEGRAM_OTEL_TRACES_EXPORTER: otlp
|
||||
TELEGRAM_OTEL_METRICS_EXPORTER: otlp
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4317
|
||||
OTEL_EXPORTER_OTLP_INSECURE: "true"
|
||||
GOMAXPROCS: "1"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 128M
|
||||
networks: [internal]
|
||||
|
||||
# --- Telegram bot (egress via the VPN sidecar in test; dials the gateway) ---
|
||||
vpn:
|
||||
container_name: scrabble-telegram-vpn
|
||||
image: docker.iliadenisov.ru/developer/amneziawg-sidecar:latest
|
||||
@@ -195,40 +245,49 @@ services:
|
||||
internal:
|
||||
aliases: [telegram]
|
||||
|
||||
telegram:
|
||||
container_name: scrabble-telegram
|
||||
image: scrabble-telegram:latest
|
||||
bot:
|
||||
container_name: scrabble-telegram-bot
|
||||
image: scrabble-telegram-bot:latest
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: platform/telegram/Dockerfile
|
||||
target: bot
|
||||
restart: unless-stopped
|
||||
logging: *default-logging
|
||||
depends_on: [vpn]
|
||||
network_mode: "service:vpn"
|
||||
environment:
|
||||
# The bot token lives ONLY in this container (ARCHITECTURE.md §12). The connector
|
||||
# requires it at boot; an empty value leaves the Telegram side down while the rest
|
||||
# of the contour still comes up.
|
||||
# The bot token lives on the bot host (ARCHITECTURE.md §12). The bot requires it
|
||||
# at boot; an empty value leaves the bot down while the rest of the contour comes up.
|
||||
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
|
||||
TELEGRAM_GAME_CHANNEL_ID: ${TELEGRAM_GAME_CHANNEL_ID:-}
|
||||
TELEGRAM_MINIAPP_URL: ${TELEGRAM_MINIAPP_URL:?set TELEGRAM_MINIAPP_URL}
|
||||
TELEGRAM_GRPC_ADDR: ":9091"
|
||||
TELEGRAM_TEST_ENV: ${TELEGRAM_TEST_ENV:-false}
|
||||
TELEGRAM_API_BASE_URL: ${TELEGRAM_API_BASE_URL:-}
|
||||
TELEGRAM_OWNS_UPDATES: "true"
|
||||
# The bot dials the gateway bot-link over mTLS. In test it reaches the gateway by
|
||||
# its internal service name through the VPN netns (Docker resolver, off-tunnel,
|
||||
# like otelcol); in prod it is a separate host dialing the gateway's public port.
|
||||
TELEGRAM_GATEWAY_ADDR: gateway:9443
|
||||
TELEGRAM_BOTLINK_SERVER_NAME: gateway
|
||||
TELEGRAM_BOTLINK_TLS_CERT: /certs/bot.crt
|
||||
TELEGRAM_BOTLINK_TLS_KEY: /certs/bot.key
|
||||
TELEGRAM_BOTLINK_TLS_CA: /certs/ca.crt
|
||||
TELEGRAM_LOG_LEVEL: ${LOG_LEVEL:-info}
|
||||
TELEGRAM_SERVICE_NAME: scrabble-telegram
|
||||
# The connector shares the VPN sidecar's netns. Routing to the collector's
|
||||
# internal IP stays off the tunnel (connected route), but the sidecar's DNS
|
||||
# hijacks name resolution: AWG_CONF must NOT carry a `DNS=` directive, else
|
||||
# `otelcol` won't resolve ("produced zero addresses"). Without DNS= the netns
|
||||
# uses Docker's resolver, which resolves both otelcol and api.telegram.org
|
||||
# (see deploy/README.md).
|
||||
TELEGRAM_SERVICE_NAME: scrabble-telegram-bot
|
||||
# The bot shares the VPN sidecar's netns. Routing to internal IPs stays off the
|
||||
# tunnel (connected route), but the sidecar's DNS hijacks name resolution:
|
||||
# AWG_CONF must NOT carry a `DNS=` directive, else `otelcol` and `gateway` won't
|
||||
# resolve. Without DNS= the netns uses Docker's resolver, which resolves the
|
||||
# internal services and api.telegram.org (see deploy/README.md).
|
||||
TELEGRAM_OTEL_TRACES_EXPORTER: otlp
|
||||
TELEGRAM_OTEL_METRICS_EXPORTER: otlp
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4317
|
||||
OTEL_EXPORTER_OTLP_INSECURE: "true"
|
||||
# The connector is light (the stress run does not drive Telegram); one P suffices.
|
||||
# The bot is light (the stress run does not drive Telegram); one P suffices.
|
||||
GOMAXPROCS: "1"
|
||||
volumes:
|
||||
- ${SCRABBLE_CONFIG_DIR:-.}/certs:/certs:ro
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
|
||||
Reference in New Issue
Block a user