Ilia Denisov
bb71e7b1c7
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
46 lines
2.0 KiB
JSON
46 lines
2.0 KiB
JSON
{
|
|
"uid": "scrabble-bot",
|
|
"title": "Scrabble — Telegram bot",
|
|
"tags": ["scrabble"],
|
|
"timezone": "",
|
|
"schemaVersion": 39,
|
|
"version": 1,
|
|
"refresh": "30s",
|
|
"time": { "from": "now-6h", "to": "now" },
|
|
"panels": [
|
|
{
|
|
"type": "stat",
|
|
"title": "Bot connected",
|
|
"description": "botlink_connected_bots: bots currently holding the gateway bot-link (1 = healthy).",
|
|
"gridPos": { "h": 5, "w": 6, "x": 0, "y": 0 },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [{ "refId": "A", "expr": "max(botlink_connected_bots)" }]
|
|
},
|
|
{
|
|
"type": "stat",
|
|
"title": "Last Bot API OK (age)",
|
|
"description": "Seconds since the bot's most recent successful Bot API call. Grows unbounded if the bot wedges.",
|
|
"gridPos": { "h": 5, "w": 6, "x": 6, "y": 0 },
|
|
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [{ "refId": "A", "expr": "time() - max(bot_tg_last_ok_unix)" }]
|
|
},
|
|
{
|
|
"type": "timeseries",
|
|
"title": "Bot API errors/s by kind",
|
|
"description": "bot_tg_errors_total by kind: connect (getUpdates), api (other sends), rate_limited (429). 429 should stay ~0.",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [{ "refId": "A", "expr": "sum by (kind) (rate(bot_tg_errors_total[5m]))", "legendFormat": "{{kind}}" }]
|
|
},
|
|
{
|
|
"type": "timeseries",
|
|
"title": "Bot-link commands/s by result",
|
|
"description": "botlink_commands_total by result: delivered / not_delivered / dropped / error. Sustained not_delivered or error means gateway sends are failing to reach Telegram.",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 5 },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [{ "refId": "A", "expr": "sum by (result) (rate(botlink_commands_total[5m]))", "legendFormat": "{{result}}" }]
|
|
}
|
|
]
|
|
}
|