docs(architecture): observability alerting + admin-alert worker
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m6s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Failing after 1m8s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m6s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Failing after 1m8s
§11 gains the alerting layer: Grafana infra rules (scrape-down, edge error-rate/p99, host mem/disk/cpu, postgres connections, TLS cert < 20d via blackbox), noDataState=OK, and the backend admin-alert worker (coalesced email on new feedback / complaints).
This commit is contained in:
@@ -1003,6 +1003,19 @@ browser), so an open in-app session reflects it at once.
|
||||
both are surfaced on the **Scrabble — Resources** Grafana dashboard, which captures the
|
||||
stress-run resource profile. (`docker_stats` replaced cAdvisor, which on the contour
|
||||
host resolved only the root cgroup — a separate-XFS `/var/lib/docker`.)
|
||||
- **Alerting.** Grafana emails infra alerts through the shared relay (its own SMTP on the
|
||||
STARTTLS port) to `SERVICE_EMAIL`, from provisioned rules
|
||||
(`deploy/grafana/provisioning/alerting/`): a scrape-target down, the gateway's
|
||||
internal-error rate and p99 latency (`edge_request_*`), host memory/disk/CPU
|
||||
(node_exporter), Postgres connection saturation (postgres_exporter), and **TLS certificate
|
||||
expiry < 20 days** — a Caddy ACME-renewal-failure signal from a **`blackbox_exporter`**
|
||||
probe of the edge caddy (`probe_ssl_earliest_cert_expiry`). Every rule is `noDataState=OK`,
|
||||
so an absent metric never false-alerts — notably the cert probe, which has no TLS target on
|
||||
the HTTP-only contour caddy. Separately, the backend's **admin-alert worker**
|
||||
(`internal/adminalert`, started from `main`) emails the operator (`ADMIN_EMAIL`, from a
|
||||
distinct `SMTP_RELAY_ADMIN_FROM`) when new player feedback or word complaints arrive,
|
||||
**coalescing** a burst into one digest per interval; both recipients may be several
|
||||
comma-separated addresses. Both paths are inert unless configured.
|
||||
- Per-request server-side timing via gin middleware from day one (the access log
|
||||
carries method, route, status, latency and the active trace id). A
|
||||
client-measured RTT piggybacked on the next request is a later enhancement.
|
||||
|
||||
Reference in New Issue
Block a user