feat(bot): report Telegram bot Bot API health to the gateway over the bot-link
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
This commit is contained in:
@@ -277,3 +277,91 @@ groups:
|
||||
- evaluator: { type: gt, params: [1800] }
|
||||
labels: { severity: critical }
|
||||
annotations: { summary: 'No WAL archived for over 30 minutes while pg_wal keeps growing — archiving is genuinely stalled; the PITR recovery point is frozen and pg_wal will fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check`.' }
|
||||
|
||||
# Remote Telegram bot health. The bot runs on its own host and exports no telemetry of its own; the
|
||||
# gateway observes it through the bot-link (botlink_connected_bots) and the health it reports over
|
||||
# that stream (bot_tg_*). All absent/NaN-safe: before a bot ever connects the metrics are absent, so
|
||||
# noDataState OK keeps them quiet on a fresh contour. The alert email does NOT go through the bot, so
|
||||
# a bot-down alert is deliverable.
|
||||
- orgId: 1
|
||||
name: scrabble-bot
|
||||
folder: Alerts
|
||||
interval: 1m
|
||||
rules:
|
||||
- uid: bot_disconnected
|
||||
title: Telegram bot disconnected
|
||||
condition: C
|
||||
for: 5m
|
||||
noDataState: OK
|
||||
execErrState: OK
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange: { from: 300, to: 0 }
|
||||
datasourceUid: prometheus
|
||||
model: { refId: A, expr: botlink_connected_bots, instant: true }
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: A
|
||||
conditions:
|
||||
- evaluator: { type: lt, params: [1] }
|
||||
labels: { severity: critical }
|
||||
annotations: { summary: 'No Telegram bot is connected to the gateway bot-link — out-of-app push and admin sends are down. Check the bot host.' }
|
||||
|
||||
# Positive liveness: a bot is connected but its last successful Bot API call is over 5 minutes
|
||||
# old — the getUpdates long-poll returns every ~minute even when idle, so a stale stamp means the
|
||||
# bot is wedged (no errors, no traffic). Guarded by "and connected" so a mere disconnect (owned by
|
||||
# bot_disconnected) does not double-fire.
|
||||
- uid: bot_tg_stale
|
||||
title: Telegram bot not reaching the Bot API
|
||||
condition: C
|
||||
for: 5m
|
||||
noDataState: OK
|
||||
execErrState: OK
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange: { from: 600, to: 0 }
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
refId: A
|
||||
expr: (time() - bot_tg_last_ok_unix) and on() (botlink_connected_bots >= 1)
|
||||
instant: true
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: A
|
||||
conditions:
|
||||
- evaluator: { type: gt, params: [300] }
|
||||
labels: { severity: critical }
|
||||
annotations: { summary: 'A connected Telegram bot has not reached the Bot API in over 5 minutes — the update poll is wedged. Check the bot host and Telegram reachability.' }
|
||||
|
||||
# 429s should be ~never once the bot honours Retry-After; any sustained rate-limiting is a symptom
|
||||
# (a send loop, a misbehaving path) worth investigating.
|
||||
- uid: bot_tg_rate_limited
|
||||
title: Telegram bot rate-limited (429)
|
||||
condition: C
|
||||
for: 5m
|
||||
noDataState: OK
|
||||
execErrState: OK
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange: { from: 900, to: 0 }
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
refId: A
|
||||
expr: increase(bot_tg_errors_total{kind="rate_limited"}[15m])
|
||||
instant: true
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: A
|
||||
conditions:
|
||||
- evaluator: { type: gt, params: [0] }
|
||||
labels: { severity: warning }
|
||||
annotations: { summary: 'The Telegram bot is being rate-limited (HTTP 429). It should honour Retry-After, so sustained 429s point to a send loop or a hot path.' }
|
||||
|
||||
Reference in New Issue
Block a user