feat(bot): report Telegram bot Bot API health to the gateway over the bot-link
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s

The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.

- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
  the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
  stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
  when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
  and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
  than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
  single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
  bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
  429s) routed to the operator email — which does not go through the bot — plus
  a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
  Retry-After, snapshot/commit, hub last-ok monotonicity).
This commit is contained in:
Ilia Denisov
2026-07-11 12:48:06 +02:00
parent 5c1f64c7d1
commit bb71e7b1c7
13 changed files with 774 additions and 98 deletions
+6 -3
View File
@@ -3,8 +3,10 @@
# docker compose -f docker-compose.bot.yml up -d
#
# The bot egresses to the Bot API directly (no VPN sidecar) and dials the main host's
# published bot-link :9443 over mTLS. It exports no telemetry — otelcol lives on the
# main host and is unreachable from here — so observe it via `docker logs` on this host.
# published bot-link :9443 over mTLS. It exports no OTLP telemetry — otelcol lives on the
# main host and is unreachable from here — but it reports its Bot API health up the bot-link,
# which the gateway turns into metrics + alerts on the main host (docs/ARCHITECTURE.md); `docker
# logs` on this host is the local detail view.
# Values come from the prod-deploy workflow (PROD_ secrets/variables); BOT_IMAGE is the
# pushed registry tag and BOTLINK_GATEWAY_ADDR is the main host's <ip>:9443.
name: scrabble-bot
@@ -50,7 +52,8 @@ services:
TELEGRAM_BOTLINK_TLS_CA: /certs/ca.crt
TELEGRAM_LOG_LEVEL: ${LOG_LEVEL:-info}
TELEGRAM_SERVICE_NAME: scrabble-telegram-bot
# No telemetry export: otelcol is on the main host, unreachable from here.
# No OTLP export: otelcol is on the main host, unreachable from here. Bot API health rides the
# bot-link instead (the gateway exposes it as metrics); see docs/ARCHITECTURE.md.
TELEGRAM_OTEL_TRACES_EXPORTER: none
TELEGRAM_OTEL_METRICS_EXPORTER: none
GOMAXPROCS: "1"