feat(bot): report Telegram bot Bot API health to the gateway over the bot-link
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s

The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.

- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
  the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
  stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
  when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
  and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
  than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
  single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
  bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
  429s) routed to the operator email — which does not go through the bot — plus
  a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
  Retry-After, snapshot/commit, hub last-ok monotonicity).
This commit is contained in:
Ilia Denisov
2026-07-11 12:48:06 +02:00
parent 5c1f64c7d1
commit bb71e7b1c7
13 changed files with 774 additions and 98 deletions
+17 -1
View File
@@ -48,14 +48,30 @@ service BotLink {
}
// FromBot is a message the bot sends to the gateway: the opening Hello, then one
// Ack per Command.
// Ack per Command and a periodic Health report.
message FromBot {
oneof msg {
Hello hello = 1;
Ack ack = 2;
Health health = 3;
}
}
// Health is a periodic report the bot pushes up the stream so the gateway can expose the remote
// bot's Bot-API health as its own metrics — the bot exports no telemetry of its own (otelcol lives
// on the main host, unreachable from the bot), so the bot-link is its only channel out. The three
// counters are DELTAS since the previous Health: the gateway adds them to cumulative counters, so a
// report lost across a reconnect only undercounts, never corrupts. last_ok_unix is the wall-clock
// second of the bot's most recent successful Bot API response (any call, including the getUpdates
// long-poll that returns every poll cycle even when idle) — an absolute liveness stamp the gateway
// surfaces as a gauge, so a silently stalled bot (no errors, no traffic) is still caught.
message Health {
uint64 connect_failures = 1; // getUpdates long-poll failures (transport / 5xx) since the last report
uint64 api_errors = 2; // other Bot API failures (transport / 5xx) since the last report
uint64 rate_limited = 3; // Bot API 429 responses since the last report (should stay ~0)
int64 last_ok_unix = 4; // unix seconds of the last successful Bot API response (0 = none yet)
}
// ToBot is a message the gateway sends to the bot. Only Command is carried today.
message ToBot {
Command command = 1;