feat(bot): report Telegram bot Bot API health to the gateway over the bot-link
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 26s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m56s
The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
This commit is contained in:
@@ -7,19 +7,26 @@ package bot
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
tgbot "github.com/go-telegram/bot"
|
||||
"github.com/go-telegram/bot/models"
|
||||
"go.uber.org/zap"
|
||||
"golang.org/x/time/rate"
|
||||
|
||||
"scrabble/platform/telegram/internal/health"
|
||||
"scrabble/platform/telegram/internal/outbox"
|
||||
"scrabble/platform/telegram/internal/support"
|
||||
)
|
||||
|
||||
// botPollTimeout is the getUpdates long-poll timeout. It matches the go-telegram/bot default; we set
|
||||
// it explicitly only because wiring the health observer (WithHTTPClient) also sets the poll timeout.
|
||||
const botPollTimeout = time.Minute
|
||||
|
||||
// Config configures the bot wrapper.
|
||||
type Config struct {
|
||||
// Token is the Bot API token.
|
||||
@@ -54,6 +61,9 @@ type Config struct {
|
||||
// pre_checkout_query updates and handles pre_checkout / successful_payment. The runtime
|
||||
// dependencies (the validator, forwarder and outbox) are wired with SetPaymentHandlers.
|
||||
AcceptPayments bool
|
||||
// Health, when set, observes every Bot API request for health metrics (reported to the gateway
|
||||
// over the bot-link) and honours a 429's Retry-After. Nil disables the observation.
|
||||
Health *health.Reporter
|
||||
}
|
||||
|
||||
// EligibilityResolver answers whether the Telegram user identified by externalID
|
||||
@@ -159,6 +169,12 @@ func New(cfg Config, log *zap.Logger) (*Bot, error) {
|
||||
if cfg.APIBaseURL != "" {
|
||||
opts = append(opts, tgbot.WithServerURL(cfg.APIBaseURL))
|
||||
}
|
||||
if cfg.Health != nil {
|
||||
// Observe every Bot API request centrally for health metrics and the 429 back-off. The
|
||||
// client timeout leaves slack over the long-poll hold (botPollTimeout - 1s server-side).
|
||||
base := &http.Client{Timeout: botPollTimeout + 10*time.Second}
|
||||
opts = append(opts, tgbot.WithHTTPClient(botPollTimeout, cfg.Health.Wrap(base)))
|
||||
}
|
||||
api, err := tgbot.New(cfg.Token, opts...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
Reference in New Issue
Block a user