feat(bot): report Telegram bot Bot API health to the gateway over the bot-link
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
This commit is contained in:
@@ -23,6 +23,7 @@ import (
|
||||
"scrabble/platform/telegram/internal/bot"
|
||||
"scrabble/platform/telegram/internal/botlink"
|
||||
"scrabble/platform/telegram/internal/config"
|
||||
"scrabble/platform/telegram/internal/health"
|
||||
"scrabble/platform/telegram/internal/outbox"
|
||||
"scrabble/platform/telegram/internal/promobot"
|
||||
"scrabble/platform/telegram/internal/support"
|
||||
@@ -31,6 +32,10 @@ import (
|
||||
// telemetryShutdownTimeout bounds the OpenTelemetry flush during process exit.
|
||||
const telemetryShutdownTimeout = 5 * time.Second
|
||||
|
||||
// healthReportInterval is how often the bot flushes its accumulated Bot API health to the gateway
|
||||
// over the bot-link. It bounds how stale a metric can be, not how often the bot talks to Telegram.
|
||||
const healthReportInterval = 30 * time.Second
|
||||
|
||||
func main() {
|
||||
cfg, err := config.LoadBot()
|
||||
if err != nil {
|
||||
@@ -81,6 +86,12 @@ func run(ctx context.Context, cfg config.BotConfig, logger *zap.Logger) error {
|
||||
}
|
||||
}
|
||||
|
||||
// The bot exports no telemetry of its own (otelcol is on the main host, unreachable from here),
|
||||
// so its Bot API health is observed centrally and reported to the gateway over the bot-link,
|
||||
// which turns the reports into metrics. Shared between the bot (which feeds it) and the bot-link
|
||||
// client (which flushes it).
|
||||
healthReporter := health.NewReporter()
|
||||
|
||||
b, err := bot.New(bot.Config{
|
||||
Token: cfg.Token,
|
||||
APIBaseURL: cfg.APIBaseURL,
|
||||
@@ -92,6 +103,7 @@ func run(ctx context.Context, cfg config.BotConfig, logger *zap.Logger) error {
|
||||
SupportChatID: cfg.SupportChatID,
|
||||
SupportStore: supportStore,
|
||||
AcceptPayments: cfg.StarsOutboxDir != "",
|
||||
Health: healthReporter,
|
||||
}, logger)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -120,6 +132,8 @@ func run(ctx context.Context, cfg config.BotConfig, logger *zap.Logger) error {
|
||||
OwnsUpdates: cfg.OwnsUpdates,
|
||||
Creds: credentials.NewTLS(tlsCfg),
|
||||
ReconnectDelay: cfg.BotLink.ReconnectDelay,
|
||||
Health: healthReporter,
|
||||
HealthInterval: healthReportInterval,
|
||||
}, exec, logger)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
Reference in New Issue
Block a user