feat(bot): Telegram bot Bot API health over the bot-link (metrics + alerts) #245
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: developer/scrabble-game#245
Reference in New Issue
Block a user
Delete Branch "feature/bot-health-telemetry"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
The Telegram bot runs on its own host and exports no telemetry (otelcol is on the main host, unreachable) — a monitoring blind spot. This makes it observable from the main host's Grafana without opening any port on the bot or shipping OTLP: the bot relays its Bot API health up the existing bot-link, and the gateway turns it into metrics + alerts.
PR1 of 2 (the second is the edge community blocklist).
Design (as agreed)
Boot order untouched (the bot must keep working when the gateway/game is down). Health is captured centrally by wrapping the Bot API HTTP client — no per-call-site edits across ~23 sites — which cleanly gives all four signals and solves idle-liveness (the
getUpdateslong-poll returns a 2xx every poll, even with no updates, so the last-ok stamp stays fresh).Health{connect_failures, api_errors, rate_limited, last_ok_unix}added to theFromBotoneof (additive → rolling-safe across the two hosts).internal/health): the HTTP wrapper stamps liveness on any 2xx, classifies transport/5xx failures (getUpdates→ connect vs other → api) and 429s, and honours a 429's Retry-After (bounded by 30s) so the bot backs off — 429 should therefore stay ~0. A 4xx other than 429 (user blocked the bot, bad request) is a normal outcome, not counted.Healthevery 30s over a single-sender loop (a gRPC stream forbids concurrent Send); a failed flush loses nothing (commit-on-success).bot_tg_errors_total{kind}+ thebot_tg_last_ok_unixliveness gauge.Three layered signals (compose, don't overlap)
botlink_connected_bots→ bot/link/host down.bot_tg_last_ok_unixstaleness → silently wedged bot (fresh even when idle).botlink_commands_total{result}→ gateway sends failing to reach Telegram.Alerts (bot disconnected / not reaching the Bot API / sustained 429) route to the operator email — which does not go through the bot, so a bot-down alert is deliverable. A
Telegram botdashboard shows all of it.Verification (local)
gofmt/vetclean ·buf generate(committed) · all four modules build · unit tests: observer classification (2xx→liveness, poll-5xx→connect, send-5xx→api, 429→rate_limited, 4xx ignored, shutdown ignored), Retry-After parsing (header + body), snapshot/commit, hub last-ok monotonicity · full gateway + telegram suites green.Note
Additive proto → a rolling deploy is safe, but prod carries two hosts (main + bot); deploy both so the bot ships the new
Health. The bot host runsdocker-compose.bot.yml(manual).The bot runs on its own host and exports no telemetry (otelcol is unreachable from there), so it was a monitoring blind spot. Observe its Bot API health centrally by wrapping the HTTP client — one place, no per-call-site instrumentation — and relay it up the existing bot-link as a periodic Health message the gateway turns into its own metrics. - proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to the FromBot oneof (additive, backward-compatible). - bot: platform/telegram/internal/health wraps the Bot API HTTP client — it stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even when idle), classifies transport/5xx failures (getUpdates vs other) and 429s, and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other than 429 is a normal per-request outcome and is not counted. - bot-link client: flush the reporter as a Health message every 30s over a single-sender loop (a gRPC stream forbids concurrent Send). - gateway: fold each report into bot_tg_errors_total{kind} and the bot_tg_last_ok_unix liveness gauge. - grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained 429s) routed to the operator email — which does not go through the bot — plus a Telegram-bot dashboard. - docs (ARCHITECTURE, compose comments); unit tests (observer classification, Retry-After, snapshot/commit, hub last-ok monotonicity).