From 70f0f9e36a13c678454023001c29f705c3818afb Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Fri, 3 Jul 2026 15:08:57 +0200 Subject: [PATCH] docs(architecture): observability alerting + admin-alert worker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §11 gains the alerting layer: Grafana infra rules (scrape-down, edge error-rate/p99, host mem/disk/cpu, postgres connections, TLS cert < 20d via blackbox), noDataState=OK, and the backend admin-alert worker (coalesced email on new feedback / complaints). --- docs/ARCHITECTURE.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index b80bedc..5f660eb 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1003,6 +1003,19 @@ browser), so an open in-app session reflects it at once. both are surfaced on the **Scrabble — Resources** Grafana dashboard, which captures the stress-run resource profile. (`docker_stats` replaced cAdvisor, which on the contour host resolved only the root cgroup — a separate-XFS `/var/lib/docker`.) +- **Alerting.** Grafana emails infra alerts through the shared relay (its own SMTP on the + STARTTLS port) to `SERVICE_EMAIL`, from provisioned rules + (`deploy/grafana/provisioning/alerting/`): a scrape-target down, the gateway's + internal-error rate and p99 latency (`edge_request_*`), host memory/disk/CPU + (node_exporter), Postgres connection saturation (postgres_exporter), and **TLS certificate + expiry < 20 days** — a Caddy ACME-renewal-failure signal from a **`blackbox_exporter`** + probe of the edge caddy (`probe_ssl_earliest_cert_expiry`). Every rule is `noDataState=OK`, + so an absent metric never false-alerts — notably the cert probe, which has no TLS target on + the HTTP-only contour caddy. Separately, the backend's **admin-alert worker** + (`internal/adminalert`, started from `main`) emails the operator (`ADMIN_EMAIL`, from a + distinct `SMTP_RELAY_ADMIN_FROM`) when new player feedback or word complaints arrive, + **coalescing** a burst into one digest per interval; both recipients may be several + comma-separated addresses. Both paths are inert unless configured. - Per-request server-side timing via gin middleware from day one (the access log carries method, route, status, latency and the active trace id). A client-measured RTT piggybacked on the next request is a later enhancement.