feat(bot): report Telegram bot Bot API health to the gateway over the bot-link
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
This commit is contained in:
@@ -48,14 +48,30 @@ service BotLink {
|
||||
}
|
||||
|
||||
// FromBot is a message the bot sends to the gateway: the opening Hello, then one
|
||||
// Ack per Command.
|
||||
// Ack per Command and a periodic Health report.
|
||||
message FromBot {
|
||||
oneof msg {
|
||||
Hello hello = 1;
|
||||
Ack ack = 2;
|
||||
Health health = 3;
|
||||
}
|
||||
}
|
||||
|
||||
// Health is a periodic report the bot pushes up the stream so the gateway can expose the remote
|
||||
// bot's Bot-API health as its own metrics — the bot exports no telemetry of its own (otelcol lives
|
||||
// on the main host, unreachable from the bot), so the bot-link is its only channel out. The three
|
||||
// counters are DELTAS since the previous Health: the gateway adds them to cumulative counters, so a
|
||||
// report lost across a reconnect only undercounts, never corrupts. last_ok_unix is the wall-clock
|
||||
// second of the bot's most recent successful Bot API response (any call, including the getUpdates
|
||||
// long-poll that returns every poll cycle even when idle) — an absolute liveness stamp the gateway
|
||||
// surfaces as a gauge, so a silently stalled bot (no errors, no traffic) is still caught.
|
||||
message Health {
|
||||
uint64 connect_failures = 1; // getUpdates long-poll failures (transport / 5xx) since the last report
|
||||
uint64 api_errors = 2; // other Bot API failures (transport / 5xx) since the last report
|
||||
uint64 rate_limited = 3; // Bot API 429 responses since the last report (should stay ~0)
|
||||
int64 last_ok_unix = 4; // unix seconds of the last successful Bot API response (0 = none yet)
|
||||
}
|
||||
|
||||
// ToBot is a message the gateway sends to the bot. Only Command is carried today.
|
||||
message ToBot {
|
||||
Command command = 1;
|
||||
|
||||
Reference in New Issue
Block a user