feat(bot): report Telegram bot Bot API health to the gateway over the bot-link
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
This commit is contained in:
@@ -3,8 +3,10 @@
|
||||
# docker compose -f docker-compose.bot.yml up -d
|
||||
#
|
||||
# The bot egresses to the Bot API directly (no VPN sidecar) and dials the main host's
|
||||
# published bot-link :9443 over mTLS. It exports no telemetry — otelcol lives on the
|
||||
# main host and is unreachable from here — so observe it via `docker logs` on this host.
|
||||
# published bot-link :9443 over mTLS. It exports no OTLP telemetry — otelcol lives on the
|
||||
# main host and is unreachable from here — but it reports its Bot API health up the bot-link,
|
||||
# which the gateway turns into metrics + alerts on the main host (docs/ARCHITECTURE.md); `docker
|
||||
# logs` on this host is the local detail view.
|
||||
# Values come from the prod-deploy workflow (PROD_ secrets/variables); BOT_IMAGE is the
|
||||
# pushed registry tag and BOTLINK_GATEWAY_ADDR is the main host's <ip>:9443.
|
||||
name: scrabble-bot
|
||||
@@ -50,7 +52,8 @@ services:
|
||||
TELEGRAM_BOTLINK_TLS_CA: /certs/ca.crt
|
||||
TELEGRAM_LOG_LEVEL: ${LOG_LEVEL:-info}
|
||||
TELEGRAM_SERVICE_NAME: scrabble-telegram-bot
|
||||
# No telemetry export: otelcol is on the main host, unreachable from here.
|
||||
# No OTLP export: otelcol is on the main host, unreachable from here. Bot API health rides the
|
||||
# bot-link instead (the gateway exposes it as metrics); see docs/ARCHITECTURE.md.
|
||||
TELEGRAM_OTEL_TRACES_EXPORTER: none
|
||||
TELEGRAM_OTEL_METRICS_EXPORTER: none
|
||||
GOMAXPROCS: "1"
|
||||
|
||||
Reference in New Issue
Block a user