feat(bot): report Telegram bot Bot API health to the gateway over the bot-link
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s

The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.

- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
  the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
  stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
  when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
  and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
  than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
  single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
  bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
  429s) routed to the operator email — which does not go through the bot — plus
  a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
  Retry-After, snapshot/commit, hub last-ok monotonicity).
This commit is contained in:
Ilia Denisov
2026-07-11 12:48:06 +02:00
parent 5c1f64c7d1
commit bb71e7b1c7
13 changed files with 774 additions and 98 deletions
+17 -1
View File
@@ -1015,6 +1015,20 @@ answering `/start` with a URL button into the **main** bot's Mini App (`?startap
button would sign initData with the promo token); it is self-contained — no bot-link, no gateway.
Session-revocation events and cursor-based stream resume stay deferred (single-instance MVP).
Because the bot **exports no telemetry of its own** (the OTel collector is on the main host,
unreachable from the bot host), it reports its **Bot API health** up the same stream: a periodic
`Health` message (`platform/telegram/internal/health`) carries delta counts of connect failures (the
getUpdates long-poll), other API errors and 429s, plus the wall-clock second of its last successful
Bot API call. The bot observes these **centrally by wrapping the Bot API HTTP client** — one place,
no per-call-site instrumentation — and there also honours a 429's `Retry-After` (bounded) so it backs
off rather than hammering (a 429 should therefore stay ~0). The gateway folds each report into its
own metrics (`bot_tg_errors_total{kind}`, the `bot_tg_last_ok_unix` liveness gauge). The remote bot
is thus monitored from the main host's Grafana with three layered signals: the **connection gauge**
(`botlink_connected_bots`) catches a link/host/process outage, the **liveness gauge** catches a
silently wedged bot (its stamp stays fresh even when idle, since getUpdates returns every poll), and
**`botlink_commands_total{result}`** catches gateway sends failing to reach Telegram. Alerts route to
the operator email, which does not depend on the bot.
A separate **advertising-banner** channel feeds the client's one-line strip (UI_DESIGN.md),
server-driven by `internal/ads`. An operator manages **campaigns** (each one placement order) in
the admin console (`/_gm/banners`): a campaign has a show **weight** (integer percent 1..100), an
@@ -1073,7 +1087,9 @@ link — misses the event; while an add-email confirmation is pending the client
`OTEL_EXPORTER_OTLP_*` environment) exports to a collector. The Postgres pool is
instrumented with otelsql and `otelgrpc` traces the backend↔gateway push stream
and the gateway↔validator and bot-link calls; the gateway also exports
`botlink_connected_bots` and `botlink_commands_total` (by result) for the bot-link. The OTLP **Collector** (OTLP/gRPC → Prometheus
`botlink_connected_bots` and `botlink_commands_total` (by result) for the bot-link, plus the remote
bot's own Bot API health it relays over the stream — `bot_tg_errors_total` (by kind) and the
`bot_tg_last_ok_unix` liveness gauge (see the bot-link section). The OTLP **Collector** (OTLP/gRPC → Prometheus
metrics + Tempo traces), **Prometheus** (15d), **Tempo** (72h) and **Grafana**
(provisioned datasources + dashboards, behind the caddy `/_gm/grafana` Basic-Auth)
are stood up with the deploy (`deploy/`); the default exporter stays