docs(architecture): observability alerting + admin-alert worker
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m6s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Failing after 1m8s

§11 gains the alerting layer: Grafana infra rules (scrape-down, edge error-rate/p99, host
mem/disk/cpu, postgres connections, TLS cert < 20d via blackbox), noDataState=OK, and the
backend admin-alert worker (coalesced email on new feedback / complaints).
This commit is contained in:
Ilia Denisov
2026-07-03 15:08:57 +02:00
parent 55f6176538
commit 70f0f9e36a
+13
View File
@@ -1003,6 +1003,19 @@ browser), so an open in-app session reflects it at once.
both are surfaced on the **Scrabble — Resources** Grafana dashboard, which captures the
stress-run resource profile. (`docker_stats` replaced cAdvisor, which on the contour
host resolved only the root cgroup — a separate-XFS `/var/lib/docker`.)
- **Alerting.** Grafana emails infra alerts through the shared relay (its own SMTP on the
STARTTLS port) to `SERVICE_EMAIL`, from provisioned rules
(`deploy/grafana/provisioning/alerting/`): a scrape-target down, the gateway's
internal-error rate and p99 latency (`edge_request_*`), host memory/disk/CPU
(node_exporter), Postgres connection saturation (postgres_exporter), and **TLS certificate
expiry < 20 days** — a Caddy ACME-renewal-failure signal from a **`blackbox_exporter`**
probe of the edge caddy (`probe_ssl_earliest_cert_expiry`). Every rule is `noDataState=OK`,
so an absent metric never false-alerts — notably the cert probe, which has no TLS target on
the HTTP-only contour caddy. Separately, the backend's **admin-alert worker**
(`internal/adminalert`, started from `main`) emails the operator (`ADMIN_EMAIL`, from a
distinct `SMTP_RELAY_ADMIN_FROM`) when new player feedback or word complaints arrive,
**coalescing** a burst into one digest per interval; both recipients may be several
comma-separated addresses. Both paths are inert unless configured.
- Per-request server-side timing via gin middleware from day one (the access log
carries method, route, status, latency and the active trace id). A
client-measured RTT piggybacked on the next request is a later enhancement.