feat(telegram): split connector into home validator + remote bot
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s

Move all Telegram egress off the main host. The single connector held the
bot token, long-polled Telegram and answered the gateway/backend over the
trusted internal network, so the whole component (including login validation)
shared fate with its VPN sidecar. Split it into two binaries that share the
token:

- cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only,
  never calls the Bot API. The gateway dials it for Telegram auth, so game
  login is now independent of Telegram reachability.
- cmd/bot (remote): Bot API long-poll + sendMessage, the only component
  reaching Telegram. It holds no inbound port — it dials the gateway over a
  new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send
  commands the gateway pushes.

The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget
(at-most-once, dropped if no bot is connected); the backend admin broadcasts
reach a gateway-served relay that forwards them and awaits the bot's ack
(SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one
inter-service link that leaves the trusted segment; validator<->gateway and
the relay stay plaintext internal. The bot is Telegram-rate-limited.

One bot now; the gateway bot registry, an owns_updates flag and per-command
ids leave seams for N later. Webhook rejected (one URL per token, adds inbound
+ a static address).

The unified test contour runs the split (the bot keeps its VPN sidecar and
dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh,
generated in CI). The prod wiring — the bot on a separate host (no VPN), the
gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH
deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX,
Stage 18).

Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway +
backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
This commit is contained in:
Ilia Denisov
2026-06-21 00:19:07 +02:00
parent 2a8717c930
commit 6aeb529f13
42 changed files with 3073 additions and 714 deletions
+83 -48
View File
@@ -45,16 +45,23 @@ Three executables plus per-platform side-services:
and a client **board-style** setting (bonus-label
mode). The visual/interaction design system is documented in
[`UI_DESIGN.md`](UI_DESIGN.md).
- **`platform/telegram`** — the Telegram side-service (the "connector", module
`scrabble/platform/telegram`). It is the only component holding the bot token — **one
unified bot** (one token + one optional game channel, §3). It
runs a Bot API long-poll loop (Mini App launch + `/start` deep-links) and serves
a gRPC API (`pkg/proto/telegram/v1`) that `gateway` (Mini App initData validation
and out-of-app push) and `backend` (operator broadcasts) call over the
trusted internal network. Its generic delivery methods are **platform-agnostic**
(keyed by the identity `external_id`), so a future VK/MAX connector reuses them; only
initData validation is Telegram-specific. It runs in its own container, egressing to
Telegram through a VPN sidecar.
- **`platform/telegram`** — the Telegram side-service (module
`scrabble/platform/telegram`), split into two binaries that share the bot token
(**one bot**, one optional game channel, §3):
- the **validator** (`cmd/validator`) verifies Mini App initData and Login Widget
data by HMAC (the bot token is the secret) and **never reaches the Bot API**, so
it runs on the main host with no VPN. The gateway calls its gRPC API
(`pkg/proto/telegram/v1`) over the trusted internal network during Telegram auth,
so **game login is independent of Telegram reachability** (§10).
- the **bot** (`cmd/bot`) runs the Bot API long-poll (Mini App launch + `/start`
deep-links) and `sendMessage`, the only component reaching the Telegram Bot API.
It holds **no inbound port**: it dials the gateway over a reverse **mTLS bot-link**
(`pkg/proto/botlink/v1`) and executes the send commands the gateway pushes
(out-of-app push, operator broadcasts), so its egress lives on a host with native
Telegram access off the main host — a VPN sidecar in the test contour, a separate
host in prod (§12). Its delivery commands are **platform-agnostic** (keyed by the
identity `external_id`), so a future VK/MAX bot reuses them; only initData
validation is Telegram-specific.
```mermaid
flowchart LR
@@ -64,9 +71,11 @@ flowchart LR
Gateway -- in-app stream --> Client
Backend -- pgx --> Postgres[(Postgres)]
Backend -. embeds .- Solver[[scrabble-solver library]]
Gateway -- gRPC (validate initData, out-of-app push) --> Telegram[Telegram connector]
Backend -. operator broadcasts (gRPC) .-> Telegram
Telegram -- Bot API (via VPN sidecar) --> TgCloud((Telegram))
Gateway -- gRPC (validate initData) --> Validator[Telegram validator]
Bot[Telegram bot] -. dials, reverse mTLS bot-link .-> Gateway
Gateway -- send commands (out-of-app push, broadcasts) --> Bot
Backend -. operator broadcasts (gRPC relay) .-> Gateway
Bot -- Bot API --> TgCloud((Telegram))
```
The MVP runs `gateway` and `backend` as single-instance processes inside a
@@ -132,15 +141,18 @@ signing, no anti-replay crypto** (these were considered and dropped — players
arrive from a platform rather than completing a mandatory registration).
- The gateway validates the originating credential **once** — Telegram `initData`
(delegated to the connector's `ValidateInitData` RPC, which holds the bot token —
(delegated to the **validator's** `ValidateInitData` RPC, which holds the bot token —
the HMAC secret — so it never reaches the gateway), an email-code login, or a guest
bootstrap — then mints a **thin opaque server session token** (`session_id`). First
Telegram contact seeds the new account's language (from the launch `language_code`)
and display name (§4).
- **Single bot.** The connector hosts **one unified bot** (one token + one optional
game channel). `ValidateInitData` validates `initData` against that single token and
returns only the Telegram user identity — there is no per-bot "service language" and no
supported-languages set on the wire. The bot's chat messages and out-of-app push are
and display name (§4). The validator runs on the main host and never reaches the Bot
API, so login does not depend on Telegram or the remote bot being up (§10, §12).
- **Single bot.** The platform side-service runs **one bot** (one token + one optional
game channel), split into a home **validator** and a remote **bot** that share the
token. `ValidateInitData` (the validator) validates `initData` against that single
token and returns only the Telegram user identity — there is no per-bot "service
language" and no supported-languages set on the wire. The bot's chat messages and
out-of-app push are
rendered in the recipient's **interface language** (`preferred_language`, en/ru), not in
any bot-scoped language, and the friend-invite **share link** (and its caption) point at
that one bot. First Telegram contact seeds the new account's `preferred_language` from the
@@ -207,7 +219,7 @@ arrive from a platform rather than completing a mandatory registration).
payment; no purchase flow yet) is carried on the account and ORed on a merge.
- **Linking** is initiated from an authenticated profile and proves
control of the identity before attaching it: **email** through the confirm-code
flow, **Telegram** through the web **Login Widget** (validated by the connector,
flow, **Telegram** through the web **Login Widget** (validated by the validator,
HMAC under `SHA-256(bot_token)` — distinct from Mini App initData; the gateway
passes the trusted `external_id` to the backend, as for `auth.telegram`). The
request step **always** sends/accepts the proof (no pre-send "already taken"
@@ -800,18 +812,20 @@ missed while the app was hidden. **Out-of-app platform push** is a fallback
the **gateway** routes from the same firehose: for an event whose recipient has **no
live in-app stream** it resolves the backend `/internal/push-target` (their Telegram
`external_id`, the recipient's **interface language** (`preferred_language`) as the render
language, and the `notifications_in_app_only` flag). It then asks the **Telegram connector**
to deliver — through the **single bot** a
localized message with a Mini App deep-link button, only when the recipient has a Telegram
identity and has not confined notifications to the app, so the two channels never duplicate. The
connector renders the message in that language; there is no per-bot routing. The out-of-app set is
language, and the `notifications_in_app_only` flag). It then pushes a deliver command over
the **bot-link** to the remote **bot****fire-and-forget, best-effort** (dropped, with a
metric, when no bot is connected) — only when the recipient has a Telegram identity and has
not confined notifications to the app, so the two channels never duplicate. The bot renders a
localized message with a Mini App deep-link button in that language; there is no per-bot
routing. The out-of-app set is
your-turn, game-over, nudge and the **invitation** (a new invitation) / friend-request notify sub-kinds;
the connector renders the message and skips the rest — so in-app-only sub-kinds like
the bot renders the message and skips the rest — so in-app-only sub-kinds like
**invitation-update** (a response/withdrawal lobby sync) and **user-blocked/-unblocked** (a
block-state sync to the blocker) never become a platform push. Operator broadcasts
(`SendToUser` / `SendToGameChannel`, §10 admin) render in an **operator-chosen** language in
the console, sent through the same single bot. Session-revocation events and
cursor-based stream resume stay deferred (single-instance MVP).
the console; the backend calls them on the **gateway's bot-link relay**, which forwards them
to the bot and **awaits its delivery ack** (so the console still reports delivered/not).
Session-revocation events and cursor-based stream resume stay deferred (single-instance MVP).
A separate **advertising-banner** channel feeds the client's one-line strip (UI_DESIGN.md),
server-driven by `internal/ads`. An operator manages **campaigns** (each one placement order) in
@@ -845,14 +859,15 @@ edits take effect on the next `profile.get` (open/reconnect/foreground), not mid
## 11. Observability
- Structured logging with `go.uber.org/zap` (JSON). OpenTelemetry tracer and
meter providers are wired in **all three services** (backend, gateway, the
Telegram connector) through a shared `pkg/telemetry` bootstrap, env-gated per
meter providers are wired in **all services** (backend, gateway, the Telegram
validator and bot) through a shared `pkg/telemetry` bootstrap, env-gated per
service by `{BACKEND,GATEWAY,TELEGRAM}_OTEL_{TRACES,METRICS}_EXPORTER` with a
default of `none` (so no collector is required locally or in CI). `stdout` is
available for debugging; **`otlp`** (gRPC, endpoint from the standard
`OTEL_EXPORTER_OTLP_*` environment) exports to a collector. The Postgres pool is
instrumented with otelsql and `otelgrpc` traces the backend↔gateway push stream
and the gateway↔connector calls. The OTLP **Collector** (OTLP/gRPC → Prometheus
and the gateway↔validator and bot-link calls; the gateway also exports
`botlink_connected_bots` and `botlink_commands_total` (by result) for the bot-link. The OTLP **Collector** (OTLP/gRPC → Prometheus
metrics + Tempo traces), **Prometheus** (15d), **Tempo** (72h) and **Grafana**
(provisioned datasources + dashboards, behind the caddy `/_gm/grafana` Basic-Auth)
are stood up with the deploy (`deploy/`); the default exporter stays
@@ -918,18 +933,21 @@ edits take effect on the next `profile.get` (open/reconnect/foreground), not mid
| Concern | Enforced by |
| --- | --- |
| Public rate limiting / anti-abuse | gateway (per-IP public/email/admin classes, per-user authenticated class; a request body cap of `GATEWAY_MAX_BODY_BYTES`; rejections are metered, summarised to the backend and surfaced in the admin console with a conservative reversible auto-flag — §11) |
| Telegram initData validation (bot-token HMAC) | the Telegram connector; the gateway delegates it over gRPC, so the bot token lives only in the connector |
| Telegram initData validation (bot-token HMAC) | the Telegram **validator**; the gateway delegates it over gRPC, so the bot token (the HMAC secret) lives only in the validator and the bot, never in the gateway |
| Session minting; email-code / guest validation | gateway (with backend) |
| Session → `user_id` resolution, `X-User-ID` injection | gateway |
| Authorisation, ownership, state transitions | backend (`X-User-ID` is the sole identity input) |
| Manual account block (suspension) | backend: a per-request gate refuses a blocked account on every `/api/v1/user/*` route except the block-status probe with **403 `account_blocked`**; the operator blocks/unblocks from the admin console (§11) |
| User feedback gate | backend rejects a guest or a `feedback_banned` account from submitting; the **gateway** also rejects a guest's `feedback.submit` (the `Op.NonGuest` flag + `is_guest` from session resolve) with **`guest_forbidden`** before any backend call; attachments are served `nosniff` with a download disposition for non-images (§15) |
| Admin authentication | a single Basic-Auth gate on `/_gm/*`, forwarded **verbatim** to the backend's server-rendered admin console (and, in the deployed contour, routing `/_gm/grafana/*` to Grafana). In the deploy the **caddy** owns this gate (§13); a local non-caddy run uses the gateway's own `GATEWAY_ADMIN_*` proxy, which the per-IP admin limiter class guards ahead of its Basic-Auth — the caddy-fronted path has no limiter (stock caddy), an accepted gap. The backend trusts the proxy (no admin principal) and guards its state-changing POSTs with a **same-origin** check — the console's CSRF defence. No operator identity is tracked |
| backend ↔ gateway ↔ connector trust | the network (only gateway may reach backend; the connector serves unauthenticated gRPC on the internal segment) |
| backend ↔ gateway ↔ validator trust | the network (only gateway may reach backend; the validator and the gateway's admin bot-link relay serve unauthenticated gRPC on the trusted internal segment) |
| remote bot ↔ gateway (bot-link) | **mutual TLS**: a private CA signs the gateway server cert and the bot client cert, and each verifies the other. The bot dials out (no inbound port, no static IP), so the channel is guarded solely by mTLS — the bot client key is as sensitive as the token (§13) |
This is an explicit, accepted MVP risk: compromise of the gateway↔backend
network segment defeats backend authentication. Mitigated by network isolation;
mutual auth is a future hardening step.
mutual auth is a future hardening step. The **bot-link** is the exception — it
already uses mutual TLS, because it is the one inter-service link that leaves the
trusted segment (the remote bot lives off the main host).
**Manual account block (suspension).** Beyond the soft, reversible high-rate flag (§11, never a
gate), an operator can hard-block an account from the admin console — permanently or until a
@@ -977,17 +995,23 @@ routes `/_gm/grafana/*` to **Grafana** (anonymous-admin, so the one shared login
it with no per-user Grafana accounts) and the rest of `/_gm/*` to the backend-rendered
**admin console**; `/app/`, `/telegram/` and the Connect path go to the gateway; the
catch-all — notably the landing at `/` — goes to the landing container. The
**Telegram connector** runs as a separate container with **no public ingress** — it
long-polls Telegram and egresses through a VPN sidecar, answering only internal gRPC.
**Telegram validator** runs as a separate container with **no public ingress**,
answering only internal gRPC (HMAC, no Telegram egress). The **Telegram bot** holds
no inbound port either: it dials the gateway's **bot-link** (mTLS) and egresses to
Telegram — through a VPN sidecar in the test contour, from a separate host in prod.
The gateway exposes the bot-link on a dedicated mTLS gRPC listener
(`GATEWAY_BOTLINK_ADDR`, internal-only in the test contour, published in prod) plus a
plaintext relay (`GATEWAY_BOTLINK_RELAY_ADDR`) the backend admin console calls.
The full contour (`deploy/docker-compose.yml`) runs one `gateway`, one `backend`,
one Postgres, the static `landing`, the connector (+ its VPN sidecar) and the **observability stack**
one Postgres, the static `landing`, the Telegram `validator` and `bot` (+ the bot's VPN
sidecar) and the **observability stack**
OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces) and Grafana
with provisioned datasources and dashboards. All three services export OTLP to the
collector; the connector shares the VPN sidecar's netns, so its `AWG_CONF` must not
with provisioned datasources and dashboards. All services export OTLP to the
collector; the bot shares the VPN sidecar's netns, so its `AWG_CONF` must not
carry a `DNS=` directive (that would hijack resolv.conf and stop it resolving
`otelcol`; without it the netns uses Docker's resolver, which resolves both
`otelcol` and `api.telegram.org`). Inter-service traffic uses a private `internal`
`otelcol` / `gateway`; without it the netns uses Docker's resolver, which resolves
`otelcol`, `gateway` and `api.telegram.org`). Inter-service traffic uses a private `internal`
network (project-scoped DNS); only caddy joins the shared external `edge` network
(alias `scrabble`).
@@ -1001,10 +1025,21 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
private-range upstreams** (`trusted_proxies private_ranges`), so the real client IP —
used for chat-moderation logging and the gateway's per-IP rate limiting — survives the
host-caddy hop; in prod (no host caddy) public clients are untrusted and Caddy uses the
real peer, so the single config is correct and spoof-safe in both contours.
real peer, so the single config is correct and spoof-safe in both contours. The
**bot-link mTLS material** (a private CA + gateway/bot leaves, CN=`gateway`) is
generated by `deploy/gen-certs.sh` before `compose up`; the bot keeps its VPN sidecar
for Telegram egress and dials the gateway by its internal name, so the bot-link stays
on the internal network.
- **Prod**: a manual SSH deploy after `development → master`. There is no
host caddy, so the contour ships its own caddy terminating TLS — set
`CADDY_SITE_ADDRESS` to the domain and the caddy does its own ACME.
`CADDY_SITE_ADDRESS` to the domain and the caddy does its own ACME. The **bot runs
on a separate host** with native Telegram access (no VPN), deployed by SSH alongside
the main app (rolled together so the bot-link protocol versions never skew); the
gateway **publishes** the bot-link port and the certificates come from `PROD_`
secrets — a long-lived CA with leaves rotated by a scheduled job. The bot dials the
gateway's public bot-link endpoint and holds no inbound port; login is unaffected if
that host or the link is down. *(This prod wiring is the deferred final stage; the
code and the unified test contour land first — see `PRERELEASE.md`.)*
## 14. CI & branches
@@ -1022,11 +1057,11 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
branch-protection required check (`CI / gate`), so a path-skipped job never blocks
a merge.
- A gated **`deploy`** job auto-rolls the **test contour** on a PR into — or a push
to — `development` (`docker compose up -d --build` on the runner host), then probes
the gateway (`GET /`) **and the Telegram connector's liveness** (via
`docker inspect`: running, not restarting, stable restart count, with a
VPN-handshake grace period, since the connector has no public ingress and a
crash-loop is otherwise invisible). A PR into `master` is test-only; the prod
to — `development` (it generates the bot-link certs, then `docker compose up -d
--build` on the runner host), then probes the gateway (`GET /`) **and the Telegram
validator's and bot's liveness** (via `docker inspect`: running, not restarting,
stable restart count, with a VPN-handshake grace period, since neither has public
ingress and a crash-loop is otherwise invisible). A PR into `master` is test-only; the prod
deploy is the manual workflow. Secrets/variables are prefixed
`TEST_`/`PROD_` per contour.
- The engine consumes `scrabble-solver` as a **published, versioned module**
+2 -2
View File
@@ -285,7 +285,7 @@ release archive, preview the words added and removed per variant against the act
dictionary, then install — which writes the version, loads it and makes it active;
versions are immutable and games in progress keep their own), and the **pending
wordlist changes** derived from accepted complaints (which feed the offline rebuild
and are marked applied after an update). When a Telegram connector is configured an operator can also
and are marked applied after an update). When the Telegram bot channel is configured an operator can also
**message a user** (by their Telegram identity) or **post to the game channel**.
State-changing actions are protected by a same-origin check; the console tracks no
operator identity.
@@ -317,7 +317,7 @@ over-grant cannot be reversed there.
The console works a **feedback** queue too (`/_gm/feedback`): the messages players sent, filtered
**unread / read / archived** with per-user search, each shown with its sender, source, channel
(with the connector bot language — en/ru — for a Telegram message), the sender's interface
(with the bot language — en/ru — for a Telegram message), the sender's interface
language, IP and any attachment. The operator can mark a message read, **reply** to the player (delivered
in-app), archive it, delete it, or delete every message from that player — and, alongside a delete,
**bar the player from feedback** (a `feedback_banned` role, distinct from a full account block: it
+2 -2
View File
@@ -293,7 +293,7 @@ identity, их игры) и **игры** — сводка, места, запи
активной; версии неизменяемы, а идущие партии остаются на своей) и **список ожидающих
правок**, выведенный из принятых жалоб (он питает офлайн-пересборку и отмечается
применённым после обновления). Если
подключён Telegram-коннектор, оператор также может **написать пользователю** (по его
подключён Telegram-бот, оператор также может **написать пользователю** (по его
Telegram-identity) или **отправить пост в игровой канал**. Изменяющие действия
защищены проверкой same-origin; личность оператора не отслеживается.
@@ -325,7 +325,7 @@ high-rate флага. С карточки пользователя операт
Консоль ведёт и очередь **обратной связи** (`/_gm/feedback`): присланные игроками сообщения с фильтром
**непрочитанные / прочитанные / архив** и поиском по пользователю, каждое — с отправителем, источником,
каналом (и языком бота-коннектора — en/ru — для сообщения из Telegram), языком интерфейса отправителя,
каналом (и языком бота — en/ru — для сообщения из Telegram), языком интерфейса отправителя,
IP и вложением. Оператор может пометить сообщение прочитанным, **ответить** игроку (доставка
в приложение), отправить в архив, удалить или удалить все сообщения этого игрока — и вместе с удалением
**запретить игроку обратную связь** (роль `feedback_banned`, отличная от полной блокировки аккаунта: