fix(monitoring): pg_archive_stalled must not false-fire on an idle database

The "WAL archiving stalled" alert fired on prod during a quiet period. An idle Postgres
archives nothing — it never force-switches an empty WAL segment on archive_timeout — so
pg_stat_archiver_last_archive_age grows past 30 min even though archiving is perfectly
healthy (failed_count 0, no .ready segments, pg_wal flat, backups current). A false
positive: no data and no disk at risk (nothing was written, so the frozen recovery point
equals the live state).

Gate the age condition on pg_wal actually growing:

  (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB)

so it fires only when WAL is being produced but not archived (the real "pg_wal fills the
disk" danger; genuine archive_command failures are already caught by pg_archive_failing).
`and on()` bridges the two metrics' differing label sets (last_archive_age carries a server
label, the pg_wal gauge does not); delta() (not increase) suits the gauge. pg_stat_wal is
not exported by this postgres_exporter, so pg_wal size growth is the available signal.
Validated on prod with promtool: the expression is empty while idle.

Also fix the runbook command in both archive alerts' annotations: `pgbackrest check` needs
`--pg1-user=scrabble`, or it connects as role "root" (which does not exist) and aborts with
"no database found".

deploy/grafana/provisioning/alerting/rules.yaml

@@ -245,8 +245,14 @@ groups:
               conditions:
                 - evaluator: { type: gt, params: [0] }
         labels: { severity: critical }
-        annotations: { summary: 'pgBackRest archive_command is failing — PITR is degrading and pg_wal can fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble check`.' }
+        annotations: { summary: 'pgBackRest archive_command is failing — PITR is degrading and pg_wal can fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check`.' }
 
+      # Fires only when the last archive is >30 min old AND pg_wal is actually growing (WAL is being
+      # produced but not archived). A genuinely idle database archives nothing — Postgres does not
+      # force-switch an empty segment on archive_timeout — so `last_archive_age` alone false-triggers
+      # during quiet hours; the pg_wal-growth guard removes that. `and on()` bridges the differing
+      # label sets (last_archive_age has a server label, the pg_wal gauge does not); delta() (not
+      # increase) suits the pg_wal_size_bytes gauge. Real archive failures are caught by pg_archive_failing.
       - uid: pg_archive_stalled
         title: WAL archiving stalled
         condition: C
@@ -255,11 +261,11 @@ groups:
         execErrState: OK
         data:
           - refId: A
-            relativeTimeRange: { from: 300, to: 0 }
+            relativeTimeRange: { from: 2100, to: 0 }
             datasourceUid: prometheus
             model:
               refId: A
-              expr: pg_stat_archiver_last_archive_age
+              expr: (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16777216)
               instant: true
           - refId: C
             datasourceUid: __expr__
@@ -270,4 +276,4 @@ groups:
               conditions:
                 - evaluator: { type: gt, params: [1800] }
         labels: { severity: critical }
-        annotations: { summary: 'No WAL segment archived for over 30 minutes (archive_timeout is 5m) — archiving is stalled; the PITR recovery point is frozen and pg_wal may grow.' }
+        annotations: { summary: 'No WAL archived for over 30 minutes while pg_wal keeps growing — archiving is genuinely stalled; the PITR recovery point is frozen and pg_wal will fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check`.' }