feat(telegram): split connector into home validator + remote bot
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s
Move all Telegram egress off the main host. The single connector held the bot token, long-polled Telegram and answered the gateway/backend over the trusted internal network, so the whole component (including login validation) shared fate with its VPN sidecar. Split it into two binaries that share the token: - cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only, never calls the Bot API. The gateway dials it for Telegram auth, so game login is now independent of Telegram reachability. - cmd/bot (remote): Bot API long-poll + sendMessage, the only component reaching Telegram. It holds no inbound port — it dials the gateway over a new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send commands the gateway pushes. The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget (at-most-once, dropped if no bot is connected); the backend admin broadcasts reach a gateway-served relay that forwards them and awaits the bot's ack (SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one inter-service link that leaves the trusted segment; validator<->gateway and the relay stay plaintext internal. The bot is Telegram-rate-limited. One bot now; the gateway bot registry, an owns_updates flag and per-command ids leave seams for N later. Webhook rejected (one URL per token, adds inbound + a static address). The unified test contour runs the split (the bot keeps its VPN sidecar and dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh, generated in CI). The prod wiring — the bot on a separate host (no VPN), the gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX, Stage 18). Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway + backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
This commit is contained in:
+83
-48
@@ -45,16 +45,23 @@ Three executables plus per-platform side-services:
|
||||
and a client **board-style** setting (bonus-label
|
||||
mode). The visual/interaction design system is documented in
|
||||
[`UI_DESIGN.md`](UI_DESIGN.md).
|
||||
- **`platform/telegram`** — the Telegram side-service (the "connector", module
|
||||
`scrabble/platform/telegram`). It is the only component holding the bot token — **one
|
||||
unified bot** (one token + one optional game channel, §3). It
|
||||
runs a Bot API long-poll loop (Mini App launch + `/start` deep-links) and serves
|
||||
a gRPC API (`pkg/proto/telegram/v1`) that `gateway` (Mini App initData validation
|
||||
and out-of-app push) and `backend` (operator broadcasts) call over the
|
||||
trusted internal network. Its generic delivery methods are **platform-agnostic**
|
||||
(keyed by the identity `external_id`), so a future VK/MAX connector reuses them; only
|
||||
initData validation is Telegram-specific. It runs in its own container, egressing to
|
||||
Telegram through a VPN sidecar.
|
||||
- **`platform/telegram`** — the Telegram side-service (module
|
||||
`scrabble/platform/telegram`), split into two binaries that share the bot token
|
||||
(**one bot**, one optional game channel, §3):
|
||||
- the **validator** (`cmd/validator`) verifies Mini App initData and Login Widget
|
||||
data by HMAC (the bot token is the secret) and **never reaches the Bot API**, so
|
||||
it runs on the main host with no VPN. The gateway calls its gRPC API
|
||||
(`pkg/proto/telegram/v1`) over the trusted internal network during Telegram auth,
|
||||
so **game login is independent of Telegram reachability** (§10).
|
||||
- the **bot** (`cmd/bot`) runs the Bot API long-poll (Mini App launch + `/start`
|
||||
deep-links) and `sendMessage`, the only component reaching the Telegram Bot API.
|
||||
It holds **no inbound port**: it dials the gateway over a reverse **mTLS bot-link**
|
||||
(`pkg/proto/botlink/v1`) and executes the send commands the gateway pushes
|
||||
(out-of-app push, operator broadcasts), so its egress lives on a host with native
|
||||
Telegram access off the main host — a VPN sidecar in the test contour, a separate
|
||||
host in prod (§12). Its delivery commands are **platform-agnostic** (keyed by the
|
||||
identity `external_id`), so a future VK/MAX bot reuses them; only initData
|
||||
validation is Telegram-specific.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
@@ -64,9 +71,11 @@ flowchart LR
|
||||
Gateway -- in-app stream --> Client
|
||||
Backend -- pgx --> Postgres[(Postgres)]
|
||||
Backend -. embeds .- Solver[[scrabble-solver library]]
|
||||
Gateway -- gRPC (validate initData, out-of-app push) --> Telegram[Telegram connector]
|
||||
Backend -. operator broadcasts (gRPC) .-> Telegram
|
||||
Telegram -- Bot API (via VPN sidecar) --> TgCloud((Telegram))
|
||||
Gateway -- gRPC (validate initData) --> Validator[Telegram validator]
|
||||
Bot[Telegram bot] -. dials, reverse mTLS bot-link .-> Gateway
|
||||
Gateway -- send commands (out-of-app push, broadcasts) --> Bot
|
||||
Backend -. operator broadcasts (gRPC relay) .-> Gateway
|
||||
Bot -- Bot API --> TgCloud((Telegram))
|
||||
```
|
||||
|
||||
The MVP runs `gateway` and `backend` as single-instance processes inside a
|
||||
@@ -132,15 +141,18 @@ signing, no anti-replay crypto** (these were considered and dropped — players
|
||||
arrive from a platform rather than completing a mandatory registration).
|
||||
|
||||
- The gateway validates the originating credential **once** — Telegram `initData`
|
||||
(delegated to the connector's `ValidateInitData` RPC, which holds the bot token —
|
||||
(delegated to the **validator's** `ValidateInitData` RPC, which holds the bot token —
|
||||
the HMAC secret — so it never reaches the gateway), an email-code login, or a guest
|
||||
bootstrap — then mints a **thin opaque server session token** (`session_id`). First
|
||||
Telegram contact seeds the new account's language (from the launch `language_code`)
|
||||
and display name (§4).
|
||||
- **Single bot.** The connector hosts **one unified bot** (one token + one optional
|
||||
game channel). `ValidateInitData` validates `initData` against that single token and
|
||||
returns only the Telegram user identity — there is no per-bot "service language" and no
|
||||
supported-languages set on the wire. The bot's chat messages and out-of-app push are
|
||||
and display name (§4). The validator runs on the main host and never reaches the Bot
|
||||
API, so login does not depend on Telegram or the remote bot being up (§10, §12).
|
||||
- **Single bot.** The platform side-service runs **one bot** (one token + one optional
|
||||
game channel), split into a home **validator** and a remote **bot** that share the
|
||||
token. `ValidateInitData` (the validator) validates `initData` against that single
|
||||
token and returns only the Telegram user identity — there is no per-bot "service
|
||||
language" and no supported-languages set on the wire. The bot's chat messages and
|
||||
out-of-app push are
|
||||
rendered in the recipient's **interface language** (`preferred_language`, en/ru), not in
|
||||
any bot-scoped language, and the friend-invite **share link** (and its caption) point at
|
||||
that one bot. First Telegram contact seeds the new account's `preferred_language` from the
|
||||
@@ -207,7 +219,7 @@ arrive from a platform rather than completing a mandatory registration).
|
||||
payment; no purchase flow yet) is carried on the account and ORed on a merge.
|
||||
- **Linking** is initiated from an authenticated profile and proves
|
||||
control of the identity before attaching it: **email** through the confirm-code
|
||||
flow, **Telegram** through the web **Login Widget** (validated by the connector,
|
||||
flow, **Telegram** through the web **Login Widget** (validated by the validator,
|
||||
HMAC under `SHA-256(bot_token)` — distinct from Mini App initData; the gateway
|
||||
passes the trusted `external_id` to the backend, as for `auth.telegram`). The
|
||||
request step **always** sends/accepts the proof (no pre-send "already taken"
|
||||
@@ -800,18 +812,20 @@ missed while the app was hidden. **Out-of-app platform push** is a fallback
|
||||
the **gateway** routes from the same firehose: for an event whose recipient has **no
|
||||
live in-app stream** it resolves the backend `/internal/push-target` (their Telegram
|
||||
`external_id`, the recipient's **interface language** (`preferred_language`) as the render
|
||||
language, and the `notifications_in_app_only` flag). It then asks the **Telegram connector**
|
||||
to deliver — through the **single bot** — a
|
||||
localized message with a Mini App deep-link button, only when the recipient has a Telegram
|
||||
identity and has not confined notifications to the app, so the two channels never duplicate. The
|
||||
connector renders the message in that language; there is no per-bot routing. The out-of-app set is
|
||||
language, and the `notifications_in_app_only` flag). It then pushes a deliver command over
|
||||
the **bot-link** to the remote **bot** — **fire-and-forget, best-effort** (dropped, with a
|
||||
metric, when no bot is connected) — only when the recipient has a Telegram identity and has
|
||||
not confined notifications to the app, so the two channels never duplicate. The bot renders a
|
||||
localized message with a Mini App deep-link button in that language; there is no per-bot
|
||||
routing. The out-of-app set is
|
||||
your-turn, game-over, nudge and the **invitation** (a new invitation) / friend-request notify sub-kinds;
|
||||
the connector renders the message and skips the rest — so in-app-only sub-kinds like
|
||||
the bot renders the message and skips the rest — so in-app-only sub-kinds like
|
||||
**invitation-update** (a response/withdrawal lobby sync) and **user-blocked/-unblocked** (a
|
||||
block-state sync to the blocker) never become a platform push. Operator broadcasts
|
||||
(`SendToUser` / `SendToGameChannel`, §10 admin) render in an **operator-chosen** language in
|
||||
the console, sent through the same single bot. Session-revocation events and
|
||||
cursor-based stream resume stay deferred (single-instance MVP).
|
||||
the console; the backend calls them on the **gateway's bot-link relay**, which forwards them
|
||||
to the bot and **awaits its delivery ack** (so the console still reports delivered/not).
|
||||
Session-revocation events and cursor-based stream resume stay deferred (single-instance MVP).
|
||||
|
||||
A separate **advertising-banner** channel feeds the client's one-line strip (UI_DESIGN.md),
|
||||
server-driven by `internal/ads`. An operator manages **campaigns** (each one placement order) in
|
||||
@@ -845,14 +859,15 @@ edits take effect on the next `profile.get` (open/reconnect/foreground), not mid
|
||||
## 11. Observability
|
||||
|
||||
- Structured logging with `go.uber.org/zap` (JSON). OpenTelemetry tracer and
|
||||
meter providers are wired in **all three services** (backend, gateway, the
|
||||
Telegram connector) through a shared `pkg/telemetry` bootstrap, env-gated per
|
||||
meter providers are wired in **all services** (backend, gateway, the Telegram
|
||||
validator and bot) through a shared `pkg/telemetry` bootstrap, env-gated per
|
||||
service by `{BACKEND,GATEWAY,TELEGRAM}_OTEL_{TRACES,METRICS}_EXPORTER` with a
|
||||
default of `none` (so no collector is required locally or in CI). `stdout` is
|
||||
available for debugging; **`otlp`** (gRPC, endpoint from the standard
|
||||
`OTEL_EXPORTER_OTLP_*` environment) exports to a collector. The Postgres pool is
|
||||
instrumented with otelsql and `otelgrpc` traces the backend↔gateway push stream
|
||||
and the gateway↔connector calls. The OTLP **Collector** (OTLP/gRPC → Prometheus
|
||||
and the gateway↔validator and bot-link calls; the gateway also exports
|
||||
`botlink_connected_bots` and `botlink_commands_total` (by result) for the bot-link. The OTLP **Collector** (OTLP/gRPC → Prometheus
|
||||
metrics + Tempo traces), **Prometheus** (15d), **Tempo** (72h) and **Grafana**
|
||||
(provisioned datasources + dashboards, behind the caddy `/_gm/grafana` Basic-Auth)
|
||||
are stood up with the deploy (`deploy/`); the default exporter stays
|
||||
@@ -918,18 +933,21 @@ edits take effect on the next `profile.get` (open/reconnect/foreground), not mid
|
||||
| Concern | Enforced by |
|
||||
| --- | --- |
|
||||
| Public rate limiting / anti-abuse | gateway (per-IP public/email/admin classes, per-user authenticated class; a request body cap of `GATEWAY_MAX_BODY_BYTES`; rejections are metered, summarised to the backend and surfaced in the admin console with a conservative reversible auto-flag — §11) |
|
||||
| Telegram initData validation (bot-token HMAC) | the Telegram connector; the gateway delegates it over gRPC, so the bot token lives only in the connector |
|
||||
| Telegram initData validation (bot-token HMAC) | the Telegram **validator**; the gateway delegates it over gRPC, so the bot token (the HMAC secret) lives only in the validator and the bot, never in the gateway |
|
||||
| Session minting; email-code / guest validation | gateway (with backend) |
|
||||
| Session → `user_id` resolution, `X-User-ID` injection | gateway |
|
||||
| Authorisation, ownership, state transitions | backend (`X-User-ID` is the sole identity input) |
|
||||
| Manual account block (suspension) | backend: a per-request gate refuses a blocked account on every `/api/v1/user/*` route except the block-status probe with **403 `account_blocked`**; the operator blocks/unblocks from the admin console (§11) |
|
||||
| User feedback gate | backend rejects a guest or a `feedback_banned` account from submitting; the **gateway** also rejects a guest's `feedback.submit` (the `Op.NonGuest` flag + `is_guest` from session resolve) with **`guest_forbidden`** before any backend call; attachments are served `nosniff` with a download disposition for non-images (§15) |
|
||||
| Admin authentication | a single Basic-Auth gate on `/_gm/*`, forwarded **verbatim** to the backend's server-rendered admin console (and, in the deployed contour, routing `/_gm/grafana/*` to Grafana). In the deploy the **caddy** owns this gate (§13); a local non-caddy run uses the gateway's own `GATEWAY_ADMIN_*` proxy, which the per-IP admin limiter class guards ahead of its Basic-Auth — the caddy-fronted path has no limiter (stock caddy), an accepted gap. The backend trusts the proxy (no admin principal) and guards its state-changing POSTs with a **same-origin** check — the console's CSRF defence. No operator identity is tracked |
|
||||
| backend ↔ gateway ↔ connector trust | the network (only gateway may reach backend; the connector serves unauthenticated gRPC on the internal segment) |
|
||||
| backend ↔ gateway ↔ validator trust | the network (only gateway may reach backend; the validator and the gateway's admin bot-link relay serve unauthenticated gRPC on the trusted internal segment) |
|
||||
| remote bot ↔ gateway (bot-link) | **mutual TLS**: a private CA signs the gateway server cert and the bot client cert, and each verifies the other. The bot dials out (no inbound port, no static IP), so the channel is guarded solely by mTLS — the bot client key is as sensitive as the token (§13) |
|
||||
|
||||
This is an explicit, accepted MVP risk: compromise of the gateway↔backend
|
||||
network segment defeats backend authentication. Mitigated by network isolation;
|
||||
mutual auth is a future hardening step.
|
||||
mutual auth is a future hardening step. The **bot-link** is the exception — it
|
||||
already uses mutual TLS, because it is the one inter-service link that leaves the
|
||||
trusted segment (the remote bot lives off the main host).
|
||||
|
||||
**Manual account block (suspension).** Beyond the soft, reversible high-rate flag (§11, never a
|
||||
gate), an operator can hard-block an account from the admin console — permanently or until a
|
||||
@@ -977,17 +995,23 @@ routes `/_gm/grafana/*` to **Grafana** (anonymous-admin, so the one shared login
|
||||
it with no per-user Grafana accounts) and the rest of `/_gm/*` to the backend-rendered
|
||||
**admin console**; `/app/`, `/telegram/` and the Connect path go to the gateway; the
|
||||
catch-all — notably the landing at `/` — goes to the landing container. The
|
||||
**Telegram connector** runs as a separate container with **no public ingress** — it
|
||||
long-polls Telegram and egresses through a VPN sidecar, answering only internal gRPC.
|
||||
**Telegram validator** runs as a separate container with **no public ingress**,
|
||||
answering only internal gRPC (HMAC, no Telegram egress). The **Telegram bot** holds
|
||||
no inbound port either: it dials the gateway's **bot-link** (mTLS) and egresses to
|
||||
Telegram — through a VPN sidecar in the test contour, from a separate host in prod.
|
||||
The gateway exposes the bot-link on a dedicated mTLS gRPC listener
|
||||
(`GATEWAY_BOTLINK_ADDR`, internal-only in the test contour, published in prod) plus a
|
||||
plaintext relay (`GATEWAY_BOTLINK_RELAY_ADDR`) the backend admin console calls.
|
||||
|
||||
The full contour (`deploy/docker-compose.yml`) runs one `gateway`, one `backend`,
|
||||
one Postgres, the static `landing`, the connector (+ its VPN sidecar) and the **observability stack** —
|
||||
one Postgres, the static `landing`, the Telegram `validator` and `bot` (+ the bot's VPN
|
||||
sidecar) and the **observability stack** —
|
||||
OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces) and Grafana
|
||||
with provisioned datasources and dashboards. All three services export OTLP to the
|
||||
collector; the connector shares the VPN sidecar's netns, so its `AWG_CONF` must not
|
||||
with provisioned datasources and dashboards. All services export OTLP to the
|
||||
collector; the bot shares the VPN sidecar's netns, so its `AWG_CONF` must not
|
||||
carry a `DNS=` directive (that would hijack resolv.conf and stop it resolving
|
||||
`otelcol`; without it the netns uses Docker's resolver, which resolves both
|
||||
`otelcol` and `api.telegram.org`). Inter-service traffic uses a private `internal`
|
||||
`otelcol` / `gateway`; without it the netns uses Docker's resolver, which resolves
|
||||
`otelcol`, `gateway` and `api.telegram.org`). Inter-service traffic uses a private `internal`
|
||||
network (project-scoped DNS); only caddy joins the shared external `edge` network
|
||||
(alias `scrabble`).
|
||||
|
||||
@@ -1001,10 +1025,21 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
|
||||
private-range upstreams** (`trusted_proxies private_ranges`), so the real client IP —
|
||||
used for chat-moderation logging and the gateway's per-IP rate limiting — survives the
|
||||
host-caddy hop; in prod (no host caddy) public clients are untrusted and Caddy uses the
|
||||
real peer, so the single config is correct and spoof-safe in both contours.
|
||||
real peer, so the single config is correct and spoof-safe in both contours. The
|
||||
**bot-link mTLS material** (a private CA + gateway/bot leaves, CN=`gateway`) is
|
||||
generated by `deploy/gen-certs.sh` before `compose up`; the bot keeps its VPN sidecar
|
||||
for Telegram egress and dials the gateway by its internal name, so the bot-link stays
|
||||
on the internal network.
|
||||
- **Prod**: a manual SSH deploy after `development → master`. There is no
|
||||
host caddy, so the contour ships its own caddy terminating TLS — set
|
||||
`CADDY_SITE_ADDRESS` to the domain and the caddy does its own ACME.
|
||||
`CADDY_SITE_ADDRESS` to the domain and the caddy does its own ACME. The **bot runs
|
||||
on a separate host** with native Telegram access (no VPN), deployed by SSH alongside
|
||||
the main app (rolled together so the bot-link protocol versions never skew); the
|
||||
gateway **publishes** the bot-link port and the certificates come from `PROD_`
|
||||
secrets — a long-lived CA with leaves rotated by a scheduled job. The bot dials the
|
||||
gateway's public bot-link endpoint and holds no inbound port; login is unaffected if
|
||||
that host or the link is down. *(This prod wiring is the deferred final stage; the
|
||||
code and the unified test contour land first — see `PRERELEASE.md`.)*
|
||||
|
||||
## 14. CI & branches
|
||||
|
||||
@@ -1022,11 +1057,11 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
|
||||
branch-protection required check (`CI / gate`), so a path-skipped job never blocks
|
||||
a merge.
|
||||
- A gated **`deploy`** job auto-rolls the **test contour** on a PR into — or a push
|
||||
to — `development` (`docker compose up -d --build` on the runner host), then probes
|
||||
the gateway (`GET /`) **and the Telegram connector's liveness** (via
|
||||
`docker inspect`: running, not restarting, stable restart count, with a
|
||||
VPN-handshake grace period, since the connector has no public ingress and a
|
||||
crash-loop is otherwise invisible). A PR into `master` is test-only; the prod
|
||||
to — `development` (it generates the bot-link certs, then `docker compose up -d
|
||||
--build` on the runner host), then probes the gateway (`GET /`) **and the Telegram
|
||||
validator's and bot's liveness** (via `docker inspect`: running, not restarting,
|
||||
stable restart count, with a VPN-handshake grace period, since neither has public
|
||||
ingress and a crash-loop is otherwise invisible). A PR into `master` is test-only; the prod
|
||||
deploy is the manual workflow. Secrets/variables are prefixed
|
||||
`TEST_`/`PROD_` per contour.
|
||||
- The engine consumes `scrabble-solver` as a **published, versioned module**
|
||||
|
||||
Reference in New Issue
Block a user