Refuse a client whose IP is in a curated CIDR feed with 403 in the same
abuseGuard, before the fail2ban ban. Prod-only (keys by real client IP), off by
default.
- ratelimit.Blocklist: a sorted-range IPv4 matcher (binary search) with an
allowlist checked first; ParseDROP reads the feed; ApplyRefresh keeps the
last-good feed on a transient fetch failure and drops it fail-open once stale
(better to under-block than block a legitimate client on a frozen feed). A
separate static CIDR set, not the per-IP fail2ban store.
- gateway: a refresher goroutine re-fetches every few hours (bounded fetch + size
cap); config GATEWAY_BLOCKLIST_{ENABLED,URL,ALLOW,REFRESH,MAX_STALENESS}.
IPv6 is not matched (a v6 client is still covered by fail2ban / the honeypot).
- observability: gateway_blocklist_blocked_total + entries/age gauges; a Grafana
alert warns before the feed is dropped; service-overview panels.
- deploy: compose env + write-prod-env.sh + prod-deploy/rollback wiring (opt-in
via PROD_ vars).
- docs (ARCHITECTURE, deploy/README); unit tests (match, allowlist, IPv6 skip,
parser, fault-tolerance) + a 403 abuse-guard integration test.
The bot runs on its own host and exports no telemetry (otelcol is unreachable
from there), so it was a monitoring blind spot. Observe its Bot API health
centrally by wrapping the HTTP client — one place, no per-call-site
instrumentation — and relay it up the existing bot-link as a periodic Health
message the gateway turns into its own metrics.
- proto: add Health (delta connect / api / 429 counters + a last-ok stamp) to
the FromBot oneof (additive, backward-compatible).
- bot: platform/telegram/internal/health wraps the Bot API HTTP client — it
stamps liveness on any 2xx (so the getUpdates long-poll keeps it fresh even
when idle), classifies transport/5xx failures (getUpdates vs other) and 429s,
and honours a 429's Retry-After (bounded) so the bot backs off; a 4xx other
than 429 is a normal per-request outcome and is not counted.
- bot-link client: flush the reporter as a Health message every 30s over a
single-sender loop (a gRPC stream forbids concurrent Send).
- gateway: fold each report into bot_tg_errors_total{kind} and the
bot_tg_last_ok_unix liveness gauge.
- grafana: alerts (bot disconnected, bot not reaching the Bot API, sustained
429s) routed to the operator email — which does not go through the bot — plus
a Telegram-bot dashboard.
- docs (ARCHITECTURE, compose comments); unit tests (observer classification,
Retry-After, snapshot/commit, hub last-ok monotonicity).
The "WAL archiving stalled" alert fired on prod during a quiet period. An idle Postgres
archives nothing — it never force-switches an empty WAL segment on archive_timeout — so
pg_stat_archiver_last_archive_age grows past 30 min even though archiving is perfectly
healthy (failed_count 0, no .ready segments, pg_wal flat, backups current). A false
positive: no data and no disk at risk (nothing was written, so the frozen recovery point
equals the live state).
Gate the age condition on pg_wal actually growing:
(pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB)
so it fires only when WAL is being produced but not archived (the real "pg_wal fills the
disk" danger; genuine archive_command failures are already caught by pg_archive_failing).
`and on()` bridges the two metrics' differing label sets (last_archive_age carries a server
label, the pg_wal gauge does not); delta() (not increase) suits the gauge. pg_stat_wal is
not exported by this postgres_exporter, so pg_wal size growth is the available signal.
Validated on prod with promtool: the expression is empty while idle.
Also fix the runbook command in both archive alerts' annotations: `pgbackrest check` needs
`--pg1-user=scrabble`, or it connects as role "root" (which does not exist) and aborts with
"no database found".
Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.
- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
deploy/README.md; record the measured cost/perf assessment.
The index.html boot guard now fires a fire-and-forget beacon (POST /telemetry/unsupported)
when it turns a client away on the unsupported-engine screen, so the owner can see — on the
Users dashboard beside "app opens" — how many real clients hit it and on which engines.
- Client: navigator.sendBeacon (fetch fallback) with a localStorage dedup keyed by app version
+ reason + Chromium, so a user reopening the app is one report, not ten.
- Gateway: a new unauthenticated POST /telemetry/unsupported handler (the client never booted,
so it carries no session), per-IP public-limited and body-capped, mirroring the export-download
route. It folds the beacon into the OTel counter unsupported_engine_total {reason =
no_bigint/no_proxy/boot_error/other, chromium}, with reason allow-listed and the Chromium major
reduced to a bounded range (normalizeUnsupported) so a spoofed beacon cannot inflate the metric
cardinality; the full user agent is logged, not labelled.
- Caddy: /telemetry/* added to the @gateway matcher (else it falls to the landing catch-all).
- Grafana: two panels on the Scrabble — Users dashboard (by reason, by Chromium major).
- Docs: ARCHITECTURE.md §11.
Tests: recordUnsupportedEngine (metric split), normalizeUnsupported (cardinality bounding), and
the handler route end to end (204 / 405 / counter). go build+vet+test and gofmt clean; ui
check/build/e2e green; the client beacon + dedup verified against a forced hard-gate (BigInt
removed) — one POST, deduped on reopen, correct reason/chromium/version labels.
Grafana: GF_SMTP from the shared relay (STARTTLS host:port) + alerting provisioning
(contact point → SERVICE_EMAIL, route-all policy, and rules for scrape-target down,
gateway internal-error rate + p99 latency, host mem/disk/cpu, postgres connections, and
TLS cert < 20 days). All rules noDataState=OK so an absent metric never false-alerts.
blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry) — effective
on prod (caddy terminates TLS; contour caddy is HTTP-only so the metric is absent).
Wires the new env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_),
prod-deploy (PROD_ + env.sh), .env.example and the README var table.
Measure uptake of the client-side local move-preview accelerator (§5) so
adoption can be watched before defaulting it on: app cold starts, dictionary
loads by result (fetched / cache_hit / miss) and move previews by path
(local / network — the backend load shed).
A small best-effort client beacon (POST /metrics/local-eval, session-gated)
batches counter deltas and posts them on a 60s timer and when the app is
backgrounded — never on the gameplay path: the in-app counters are plain
in-memory increments, only the periodic flush touches the network and it is
fire-and-forget. The gateway folds each batch into three OTel counters
(local_eval_cold_start_total, local_eval_dict_load_total,
local_eval_preview_total), clamped against a spoofed inflation.
- gateway: counters + recordLocalEval + session-gated /metrics/local-eval handler
- ui: localeval-metrics accumulator/beacon; hooks in the dict loader, in
Game.recompute and in bootstrap (skipped under the mock harness)
- caddy: route /metrics/* to the gateway
- docs: ARCHITECTURE §11; Grafana "Scrabble — Users" dashboard panels
The "Database size" stat had no thresholds, so Grafana applied its stat
default (green base, red at >=80). The query is pg_database_size_bytes, so a
healthy ~9 MiB database (9.4M >> 80) rendered permanently RED on the
Scrabble - Resources dashboard (test + prod), reading as a false alert; the
neighbouring percentunit cache-hit stat stayed green only because its 0..1
values fall under 80.
Add absolute byte thresholds sized to the 40 GiB prod disk (4.6 GiB used,
observability bounded -- Tempo <=1 GiB, Prometheus 7d -- so the DB is the
only unbounded grower): green up to 8 GiB, yellow at 8 GiB (~20% of disk),
red at 16 GiB (~40%), an early warning with ample runway before the disk
tightens, not a panic line. Cosmetic panel coloring only; there are no
Grafana alert rules provisioned.
Visualises the chat read-receipt metrics added with the read-receipt feature
(the dashboard was deferred there per the owner): posted rate by kind, the
unread backlog (chat_unread_messages gauge), and the publish-to-read latency
(chat_read_duration — average by kind plus overall p50/p95). Mirrors the
game-domain dashboard's structure; the file provider auto-discovers it.
Follow-ups on the honest-AI game, same PR:
- GCG export labels the robot seat "AI" instead of its pool name (ExportGCG
overrides via accounts.IsRobot); the in-app 🤖 is unchanged.
- vs_ai games emit no your_turn (the robot replies instantly, so it would be
redundant); opponent_moved still advances the UI.
- Admin console shows the AI flag: a 🤖 column in /games and an "AI game" line
on the game card (GameRow/GameDetailView gain VsAI).
- games_started_total / games_abandoned_total gain a vs_ai attribute; the
Grafana Game-domain dashboard splits started/abandoned into human and AI
panels.
Tests: metrics unit (vs_ai split); integration (no your_turn, GCG "AI").
Observability: replace cAdvisor (which resolves only the root cgroup on the
contour host — separate-XFS /var/lib/docker) with the otelcol docker_stats
receiver, which reads per-container CPU/memory/network straight from the Docker
API and works the same in prod. The collector joins the host docker group
(DOCKER_GID, default 989) and mounts the socket read-only; its metrics flow out
through the existing prometheus exporter, so the cAdvisor scrape job and the
privileged cAdvisor service are removed. The Resources dashboard panels are
retargeted to the docker_stats metric names (container_name label;
container.cpu.utilization/100 == cores).
Container limits: apply deploy.resources.limits (honoured by Compose v2) across
the contour and pin GOMAXPROCS to the CPU limit on the Go services so the runtime
matches the cgroup quota. Starting values are generous over the R2 peak (~1 core /
<=100 MiB per app service) to avoid skewing or OOM-killing the measurement run;
they are tightened to the agreed prod sizing after the final stress run (R7
Round 2). The privileged VPN sidecar is left unconstrained.
New scrabble/loadtest module (the pre-release stress harness): seeds 1000 guest +
10000 durable accounts with pre-created sessions directly in Postgres (token hash
matches backend/internal/session), drives virtual players through the edge protocol
(real 2-4p games assembled via invitations, mid-ranked legal moves generated locally
by the embedded scrabble-solver — the edge carries no board, so the client replays
history), plus nudge/chat/check-word/draft/profile/stats and a gateway-hammer that
verifies the rate limiter. Prints a trip-report summary (per-op latency percentiles,
result codes, live-event tally). Go unit tests cover the pure pieces; the DAWG-backed
move test runs under BACKEND_DICT_DIR.
Contour: add cAdvisor + postgres_exporter + a 'Scrabble - Resources' Grafana
dashboard and the two Prometheus scrape jobs, for the R2/R7 stress-run resource
baseline.
CI: gate ./loadtest/... (path filter + vet/build/test). Docs: TESTING, ARCHITECTURE,
project CLAUDE repo layout.
- #10 a `changes` job path-filters unit/integration/ui; an always-running `gate` job aggregates them (success-or-skipped) and becomes the only required check
- #9 deploy adds a Telegram-connector liveness probe (docker inspect: running, not restarting, stable restart count) with a VPN-handshake grace period
- #1a Game-domain dashboard gains a 'Move think-time by phase (p50/p95)' panel
- deploy README: branch protection now requires only CI / gate
- deploy/docker-compose.yml: mount the provisioned dashboards at
/etc/grafana/dashboards, not /var/lib/grafana/dashboards — the grafana-data
volume mounts over the latter and shadows the nested bind, so the provider
logged "readdirent /var/lib/grafana/dashboards: no such file or directory".
dashboards.yaml provider path updated to match.
- Connector telemetry stays OTLP. The VPN sidecar's netns reaches the collector's
internal IP fine (connected route, off-tunnel), but the sidecar's DNS hijacks
name resolution: AWG_CONF must NOT carry a DNS= directive, else otelcol won't
resolve ("produced zero addresses"). Without DNS= the netns uses Docker's
resolver (resolves both otelcol and api.telegram.org). Documented in
deploy/README.md (AWG_CONF row + wiring note), ARCHITECTURE §13, compose comment.
- backend + gateway multi-stage distroless Dockerfiles; the gateway embeds and
serves the SPA at / and /telegram/ via go:embed (committed dist placeholder,
real build baked in by the image's node stage)
- deploy/docker-compose.yml: backend + gateway + Postgres + Telegram connector
(VPN sidecar) + OTel Collector + Prometheus (15d) + Tempo (72h) + Grafana,
fronted by a caddy owning a single /_gm Basic-Auth (admin console + Grafana
subpath); inter-service on a private network, only caddy on the edge network
- new metrics: backend accounts_created_total{kind} (robots excluded) and an
in-memory gateway active_users{window=24h,7d} gauge
- CI: single .gitea/workflows/ci.yaml (unit/integration/ui + a gated test-contour
deploy) on the new feature/* -> development -> master branch model; the old
go-unit/integration/ui-test workflows are folded in; the connector-scoped
compose is retired (superseded by deploy/)
- docs: ARCHITECTURE §11/§12/§13, root + gateway READMEs, CLAUDE.md branching,
PLAN.md (stage 16 done + refinements + Stage 17 forward-notes)
Ilia Denisov