feat(observability): Grafana infra alerts + admin email notifications (PR4) #165

Merged

developer merged 5 commits from feature/email-relay-pr4 into development 2026-07-03 14:05:45 +00:00

Conversation 1 Commits 5 Files Changed 21

+675 -14

developer commented 2026-07-03 13:09:48 +00:00

Owner

Copy Link

Copy Source

PR4 — Grafana infra alerts + admin email notifications (final email-relay layer)

Backend admin alerts

• New internal/adminalert worker: polls for player feedback + word complaints arriving
  since the last check and coalesces a burst into one digest email per interval (5 min),
  to ADMIN_EMAIL from a distinct SMTP_RELAY_ADMIN_FROM. Inert unless both are set.
• The mailer gains a per-message From override and splits a comma-separated To into
  separate recipients (go-mail needs a list; several operator mailboxes in one message).
• feedback/game stores gain CountSince / CountComplaintsSince.

Grafana infra alerts

• GF_SMTP from the shared relay on its STARTTLS host:port (GRAFANA_SMTP_HOST; Grafana
  speaks STARTTLS, not the backend's implicit TLS), reusing the relay creds + SERVICE_FROM.
• Provisioned alerting (deploy/grafana/provisioning/alerting/): a contact point →
  SERVICE_EMAIL, a route-all policy, and rules for scrape-target down, gateway
  internal-error rate + p99 latency, host mem/disk/CPU, Postgres connections, and TLS cert
  < 20 days. Every rule is noDataState=OK so an absent metric never false-alerts.
• Metric names verified against the live contour Prometheus (edge_request_*, node_*,
  pg_*, probe_ssl_*). result="internal" is the 5xx-equivalent edge outcome.

Cert-expiry

• New blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry)
  → the cert-renewal-failure alert. Effective on prod (caddy terminates TLS on :443);
  on the test contour caddy is HTTP-only, so the metric is absent and the rule no-ops.
  Threshold 20 days (Caddy renews at ~30d, so 20 gives lead time to react).

Wiring & docs

• Env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_*),
  prod-deploy (PROD_* + env.sh), .env.example, README var table; ARCHITECTURE §11.
• New CI vars (owner sets): TEST/PROD_{SMTP_RELAY_ADMIN_FROM, ADMIN_EMAIL, SMTP_RELAY_SERVICE_FROM, SERVICE_EMAIL, GRAFANA_SMTP_HOST, GF_SMTP_ENABLED}.

Locally verified: Go build/vet/gofmt/unit (incl. adminalert), full integration package,
compose config (base + prod overlay), all new YAML parses. After the contour deploys,
worth checking docker logs scrabble-grafana for provisioning-load errors (Grafana skips
malformed alerting provisioning silently) and confirming the $__env{SERVICE_EMAIL} contact
point resolved. This is the last layer of the email-relay feature (PR1a/1b/2/3 merged).

## PR4 — Grafana infra alerts + admin email notifications (final email-relay layer) ### Backend admin alerts - New `internal/adminalert` worker: polls for player feedback + word complaints arriving since the last check and **coalesces a burst into one digest email per interval** (5 min), to `ADMIN_EMAIL` from a distinct `SMTP_RELAY_ADMIN_FROM`. Inert unless both are set. - The mailer gains a per-message **From** override and splits a **comma-separated To** into separate recipients (go-mail needs a list; several operator mailboxes in one message). - `feedback`/`game` stores gain `CountSince` / `CountComplaintsSince`. ### Grafana infra alerts - `GF_SMTP` from the shared relay on its **STARTTLS** host:port (`GRAFANA_SMTP_HOST`; Grafana speaks STARTTLS, not the backend's implicit TLS), reusing the relay creds + `SERVICE_FROM`. - Provisioned alerting (`deploy/grafana/provisioning/alerting/`): a contact point → `SERVICE_EMAIL`, a route-all policy, and rules for **scrape-target down, gateway internal-error rate + p99 latency, host mem/disk/CPU, Postgres connections, and TLS cert < 20 days**. Every rule is `noDataState=OK` so an absent metric never false-alerts. - Metric names verified against the live contour Prometheus (`edge_request_*`, `node_*`, `pg_*`, `probe_ssl_*`). `result="internal"` is the 5xx-equivalent edge outcome. ### Cert-expiry - New **`blackbox_exporter`** probes the edge caddy's TLS (`probe_ssl_earliest_cert_expiry`) → the cert-renewal-failure alert. Effective on **prod** (caddy terminates TLS on :443); on the test contour caddy is HTTP-only, so the metric is absent and the rule no-ops. Threshold **20 days** (Caddy renews at ~30d, so 20 gives lead time to react). ### Wiring & docs - Env through compose (backend admin From/To, Grafana SMTP), `ci.yaml` (`TEST_*`), `prod-deploy` (`PROD_*` + `env.sh`), `.env.example`, README var table; ARCHITECTURE §11. - New CI vars (owner sets): `TEST/PROD_{SMTP_RELAY_ADMIN_FROM, ADMIN_EMAIL, SMTP_RELAY_SERVICE_FROM, SERVICE_EMAIL, GRAFANA_SMTP_HOST, GF_SMTP_ENABLED}`. Locally verified: Go build/vet/gofmt/unit (incl. adminalert), full integration package, compose config (base + prod overlay), all new YAML parses. **After the contour deploys, worth checking `docker logs scrabble-grafana` for provisioning-load errors** (Grafana skips malformed alerting provisioning silently) and confirming the `$__env{SERVICE_EMAIL}` contact point resolved. This is the **last layer of the email-relay feature** (PR1a/1b/2/3 merged).

developer added 3 commits 2026-07-03 13:09:48 +00:00

feat(adminalert): operator email on new feedback / word complaints ... 8d2cd97e17

New adminalert worker polls for feedback + word complaints arriving since the last check
and coalesces a burst into one digest email per interval (5 min), inert unless a distinct
admin sender (BACKEND_SMTP_ADMIN_FROM) and recipient (BACKEND_ADMIN_EMAIL, comma-separated
allowed) are configured. The mailer gains a per-message From override and splits a
comma-separated To into separate recipients (go-mail needs them as a list). Feedback/game
stores gain CountSince/CountComplaintsSince. Unit tests cover the digest, the skip-when-
empty, and the recipient split.

feat(deploy): Grafana infra alerts + blackbox cert probe + admin-alert wiring ... 55f6176538

Grafana: GF_SMTP from the shared relay (STARTTLS host:port) + alerting provisioning
(contact point → SERVICE_EMAIL, route-all policy, and rules for scrape-target down,
gateway internal-error rate + p99 latency, host mem/disk/cpu, postgres connections, and
TLS cert < 20 days). All rules noDataState=OK so an absent metric never false-alerts.
blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry) — effective
on prod (caddy terminates TLS; contour caddy is HTTP-only so the metric is absent).
Wires the new env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_),
prod-deploy (PROD_ + env.sh), .env.example and the README var table.

docs(architecture): observability alerting + admin-alert worker ... 70f0f9e36a

§11 gains the alerting layer: Grafana infra rules (scrape-down, edge error-rate/p99, host
mem/disk/cpu, postgres connections, TLS cert < 20d via blackbox), noDataState=OK, and the
backend admin-alert worker (coalesced email on new feedback / complaints).

developer added 1 commit 2026-07-03 13:17:44 +00:00

fix(deploy): Grafana SMTP reuses relay host + explicit GRAFANA_SMTP_PORT ... ec1bdfca00

Drop the separate GRAFANA_SMTP_HOST (it duplicated SMTP_RELAY_HOST): Grafana differs from
the backend only in needing the STARTTLS port, so GF_SMTP_HOST is composed from
SMTP_RELAY_HOST + GRAFANA_SMTP_PORT (an explicit per-contour var, no magic default), reusing
SMTP_RELAY_USER/PASS. Wired through ci/prod/.env.example/README.

developer added 1 commit 2026-07-03 13:30:42 +00:00

fix(deploy): stage blackbox config + derive bare Grafana SMTP from-address ... 854c4b3005

Two contour-deploy failures from PR4: (1) the deploy did not stage deploy/blackbox/, so
blackbox_exporter crash-looped on a missing config mount — add blackbox to the ci.yaml cp
and the prod-deploy tar; (2) Grafana rejects the 'Name <addr>' From form the backend go-mail
accepts and validates it even when SMTP is disabled, crash-looping Grafana — the deploy now
splits SMTP_RELAY_SERVICE_FROM into a bare GF_SMTP_FROM_ADDRESS + GRAFANA_SMTP_FROM_NAME
(handles both display and bare forms). Verified the sed split + compose render.

owner approved these changes 2026-07-03 14:05:12 +00:00

developer merged commit 60faa4f064 into development 2026-07-03 14:05:45 +00:00

developer deleted branch feature/email-relay-pr4 2026-07-03 14:05:45 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

owner

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

owner (Ilia) developer

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: developer/scrabble-game#165