The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
The "WAL archiving stalled" alert fired on prod during a quiet period. An idle Postgres
archives nothing — it never force-switches an empty WAL segment on archive_timeout — so
pg_stat_archiver_last_archive_age grows past 30 min even though archiving is perfectly
healthy (failed_count 0, no .ready segments, pg_wal flat, backups current). A false
positive: no data and no disk at risk (nothing was written, so the frozen recovery point
equals the live state).
Gate the age condition on pg_wal actually growing:
(pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB)
so it fires only when WAL is being produced but not archived (the real "pg_wal fills the
disk" danger; genuine archive_command failures are already caught by pg_archive_failing).
`and on()` bridges the two metrics' differing label sets (last_archive_age carries a server
label, the pg_wal gauge does not); delta() (not increase) suits the gauge. pg_stat_wal is
not exported by this postgres_exporter, so pg_wal size growth is the available signal.
Validated on prod with promtool: the expression is empty while idle.
Also fix the runbook command in both archive alerts' annotations: `pgbackrest check` needs
`--pg1-user=scrabble`, or it connects as role "root" (which does not exist) and aborts with
"no database found".
Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.
- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
deploy/README.md; record the measured cost/perf assessment.
Grafana: GF_SMTP from the shared relay (STARTTLS host:port) + alerting provisioning
(contact point → SERVICE_EMAIL, route-all policy, and rules for scrape-target down,
gateway internal-error rate + p99 latency, host mem/disk/cpu, postgres connections, and
TLS cert < 20 days). All rules noDataState=OK so an absent metric never false-alerts.
blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry) — effective
on prod (caddy terminates TLS; contour caddy is HTTP-only so the metric is absent).
Wires the new env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_),
prod-deploy (PROD_ + env.sh), .env.example and the README var table.
- deploy/docker-compose.yml: mount the provisioned dashboards at
/etc/grafana/dashboards, not /var/lib/grafana/dashboards — the grafana-data
volume mounts over the latter and shadows the nested bind, so the provider
logged "readdirent /var/lib/grafana/dashboards: no such file or directory".
dashboards.yaml provider path updated to match.
- Connector telemetry stays OTLP. The VPN sidecar's netns reaches the collector's
internal IP fine (connected route, off-tunnel), but the sidecar's DNS hijacks
name resolution: AWG_CONF must NOT carry a DNS= directive, else otelcol won't
resolve ("produced zero addresses"). Without DNS= the netns uses Docker's
resolver (resolves both otelcol and api.telegram.org). Documented in
deploy/README.md (AWG_CONF row + wiring note), ARCHITECTURE §13, compose comment.
- backend + gateway multi-stage distroless Dockerfiles; the gateway embeds and
serves the SPA at / and /telegram/ via go:embed (committed dist placeholder,
real build baked in by the image's node stage)
- deploy/docker-compose.yml: backend + gateway + Postgres + Telegram connector
(VPN sidecar) + OTel Collector + Prometheus (15d) + Tempo (72h) + Grafana,
fronted by a caddy owning a single /_gm Basic-Auth (admin console + Grafana
subpath); inter-service on a private network, only caddy on the edge network
- new metrics: backend accounts_created_total{kind} (robots excluded) and an
in-memory gateway active_users{window=24h,7d} gauge
- CI: single .gitea/workflows/ci.yaml (unit/integration/ui + a gated test-contour
deploy) on the new feature/* -> development -> master branch model; the old
go-unit/integration/ui-test workflows are folded in; the connector-scoped
compose is retired (superseded by deploy/)
- docs: ARCHITECTURE §11/§12/§13, root + gateway READMEs, CLAUDE.md branching,
PLAN.md (stage 16 done + refinements + Stage 17 forward-notes)
Ilia Denisov