The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
The index.html boot guard now fires a fire-and-forget beacon (POST /telemetry/unsupported)
when it turns a client away on the unsupported-engine screen, so the owner can see — on the
Users dashboard beside "app opens" — how many real clients hit it and on which engines.
- Client: navigator.sendBeacon (fetch fallback) with a localStorage dedup keyed by app version
+ reason + Chromium, so a user reopening the app is one report, not ten.
- Gateway: a new unauthenticated POST /telemetry/unsupported handler (the client never booted,
so it carries no session), per-IP public-limited and body-capped, mirroring the export-download
route. It folds the beacon into the OTel counter unsupported_engine_total {reason =
no_bigint/no_proxy/boot_error/other, chromium}, with reason allow-listed and the Chromium major
reduced to a bounded range (normalizeUnsupported) so a spoofed beacon cannot inflate the metric
cardinality; the full user agent is logged, not labelled.
- Caddy: /telemetry/* added to the @gateway matcher (else it falls to the landing catch-all).
- Grafana: two panels on the Scrabble — Users dashboard (by reason, by Chromium major).
- Docs: ARCHITECTURE.md §11.
Tests: recordUnsupportedEngine (metric split), normalizeUnsupported (cardinality bounding), and
the handler route end to end (204 / 405 / counter). go build+vet+test and gofmt clean; ui
check/build/e2e green; the client beacon + dedup verified against a forced hard-gate (BigInt
removed) — one POST, deduped on reopen, correct reason/chromium/version labels.
Measure uptake of the client-side local move-preview accelerator (§5) so
adoption can be watched before defaulting it on: app cold starts, dictionary
loads by result (fetched / cache_hit / miss) and move previews by path
(local / network — the backend load shed).
A small best-effort client beacon (POST /metrics/local-eval, session-gated)
batches counter deltas and posts them on a 60s timer and when the app is
backgrounded — never on the gameplay path: the in-app counters are plain
in-memory increments, only the periodic flush touches the network and it is
fire-and-forget. The gateway folds each batch into three OTel counters
(local_eval_cold_start_total, local_eval_dict_load_total,
local_eval_preview_total), clamped against a spoofed inflation.
- gateway: counters + recordLocalEval + session-gated /metrics/local-eval handler
- ui: localeval-metrics accumulator/beacon; hooks in the dict loader, in
Game.recompute and in bootstrap (skipped under the mock harness)
- caddy: route /metrics/* to the gateway
- docs: ARCHITECTURE §11; Grafana "Scrabble — Users" dashboard panels
The "Database size" stat had no thresholds, so Grafana applied its stat
default (green base, red at >=80). The query is pg_database_size_bytes, so a
healthy ~9 MiB database (9.4M >> 80) rendered permanently RED on the
Scrabble - Resources dashboard (test + prod), reading as a false alert; the
neighbouring percentunit cache-hit stat stayed green only because its 0..1
values fall under 80.
Add absolute byte thresholds sized to the 40 GiB prod disk (4.6 GiB used,
observability bounded -- Tempo <=1 GiB, Prometheus 7d -- so the DB is the
only unbounded grower): green up to 8 GiB, yellow at 8 GiB (~20% of disk),
red at 16 GiB (~40%), an early warning with ample runway before the disk
tightens, not a panic line. Cosmetic panel coloring only; there are no
Grafana alert rules provisioned.
Visualises the chat read-receipt metrics added with the read-receipt feature
(the dashboard was deferred there per the owner): posted rate by kind, the
unread backlog (chat_unread_messages gauge), and the publish-to-read latency
(chat_read_duration — average by kind plus overall p50/p95). Mirrors the
game-domain dashboard's structure; the file provider auto-discovers it.
Follow-ups on the honest-AI game, same PR:
- GCG export labels the robot seat "AI" instead of its pool name (ExportGCG
overrides via accounts.IsRobot); the in-app 🤖 is unchanged.
- vs_ai games emit no your_turn (the robot replies instantly, so it would be
redundant); opponent_moved still advances the UI.
- Admin console shows the AI flag: a 🤖 column in /games and an "AI game" line
on the game card (GameRow/GameDetailView gain VsAI).
- games_started_total / games_abandoned_total gain a vs_ai attribute; the
Grafana Game-domain dashboard splits started/abandoned into human and AI
panels.
Tests: metrics unit (vs_ai split); integration (no your_turn, GCG "AI").
Observability: replace cAdvisor (which resolves only the root cgroup on the
contour host — separate-XFS /var/lib/docker) with the otelcol docker_stats
receiver, which reads per-container CPU/memory/network straight from the Docker
API and works the same in prod. The collector joins the host docker group
(DOCKER_GID, default 989) and mounts the socket read-only; its metrics flow out
through the existing prometheus exporter, so the cAdvisor scrape job and the
privileged cAdvisor service are removed. The Resources dashboard panels are
retargeted to the docker_stats metric names (container_name label;
container.cpu.utilization/100 == cores).
Container limits: apply deploy.resources.limits (honoured by Compose v2) across
the contour and pin GOMAXPROCS to the CPU limit on the Go services so the runtime
matches the cgroup quota. Starting values are generous over the R2 peak (~1 core /
<=100 MiB per app service) to avoid skewing or OOM-killing the measurement run;
they are tightened to the agreed prod sizing after the final stress run (R7
Round 2). The privileged VPN sidecar is left unconstrained.
New scrabble/loadtest module (the pre-release stress harness): seeds 1000 guest +
10000 durable accounts with pre-created sessions directly in Postgres (token hash
matches backend/internal/session), drives virtual players through the edge protocol
(real 2-4p games assembled via invitations, mid-ranked legal moves generated locally
by the embedded scrabble-solver — the edge carries no board, so the client replays
history), plus nudge/chat/check-word/draft/profile/stats and a gateway-hammer that
verifies the rate limiter. Prints a trip-report summary (per-op latency percentiles,
result codes, live-event tally). Go unit tests cover the pure pieces; the DAWG-backed
move test runs under BACKEND_DICT_DIR.
Contour: add cAdvisor + postgres_exporter + a 'Scrabble - Resources' Grafana
dashboard and the two Prometheus scrape jobs, for the R2/R7 stress-run resource
baseline.
CI: gate ./loadtest/... (path filter + vet/build/test). Docs: TESTING, ARCHITECTURE,
project CLAUDE repo layout.
- #10 a `changes` job path-filters unit/integration/ui; an always-running `gate` job aggregates them (success-or-skipped) and becomes the only required check
- #9 deploy adds a Telegram-connector liveness probe (docker inspect: running, not restarting, stable restart count) with a VPN-handshake grace period
- #1a Game-domain dashboard gains a 'Move think-time by phase (p50/p95)' panel
- deploy README: branch protection now requires only CI / gate
- backend + gateway multi-stage distroless Dockerfiles; the gateway embeds and
serves the SPA at / and /telegram/ via go:embed (committed dist placeholder,
real build baked in by the image's node stage)
- deploy/docker-compose.yml: backend + gateway + Postgres + Telegram connector
(VPN sidecar) + OTel Collector + Prometheus (15d) + Tempo (72h) + Grafana,
fronted by a caddy owning a single /_gm Basic-Auth (admin console + Grafana
subpath); inter-service on a private network, only caddy on the edge network
- new metrics: backend accounts_created_total{kind} (robots excluded) and an
in-memory gateway active_users{window=24h,7d} gauge
- CI: single .gitea/workflows/ci.yaml (unit/integration/ui + a gated test-contour
deploy) on the new feature/* -> development -> master branch model; the old
go-unit/integration/ui-test workflows are folded in; the connector-scoped
compose is retired (superseded by deploy/)
- docs: ARCHITECTURE §11/§12/§13, root + gateway READMEs, CLAUDE.md branching,
PLAN.md (stage 16 done + refinements + Stage 17 forward-notes)
Ilia Denisov