fix(monitoring): pg_archive_stalled must not false-fire on an idle database
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m55s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m55s
The "WAL archiving stalled" alert fired on prod during a quiet period. An idle Postgres archives nothing — it never force-switches an empty WAL segment on archive_timeout — so pg_stat_archiver_last_archive_age grows past 30 min even though archiving is perfectly healthy (failed_count 0, no .ready segments, pg_wal flat, backups current). A false positive: no data and no disk at risk (nothing was written, so the frozen recovery point equals the live state). Gate the age condition on pg_wal actually growing: (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB) so it fires only when WAL is being produced but not archived (the real "pg_wal fills the disk" danger; genuine archive_command failures are already caught by pg_archive_failing). `and on()` bridges the two metrics' differing label sets (last_archive_age carries a server label, the pg_wal gauge does not); delta() (not increase) suits the gauge. pg_stat_wal is not exported by this postgres_exporter, so pg_wal size growth is the available signal. Validated on prod with promtool: the expression is empty while idle. Also fix the runbook command in both archive alerts' annotations: `pgbackrest check` needs `--pg1-user=scrabble`, or it connects as role "root" (which does not exist) and aborts with "no database found".
This commit is contained in:
+4
-2
@@ -255,8 +255,10 @@ shipping or redeploying this stack does **not** start archiving — the artifact
|
||||
armed, which is why an un-armed prod deploy can never pile WAL onto the disk. The base-backup
|
||||
timer is provisioned by the Ansible `main` role behind `pitr_enabled` (also default off). Two
|
||||
Grafana alerts watch health: `WAL archiving failing` (`pg_stat_archiver_failed_count` rising)
|
||||
and `WAL archiving stalled` (`pg_stat_archiver_last_archive_age` over 30 min) — both
|
||||
absent/NaN-safe, so they stay quiet until archiving is armed.
|
||||
and `WAL archiving stalled` (last archive over 30 min old **while `pg_wal_size_bytes` is growing**
|
||||
— the pg_wal-growth guard keeps an idle database, which archives nothing because it writes nothing,
|
||||
from false-triggering during quiet hours) — both absent/NaN-safe, so they stay quiet until
|
||||
archiving is armed.
|
||||
|
||||
**Assessment (owner-reviewed; the gate before the first real money).** Measured on prod
|
||||
`pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At
|
||||
|
||||
Reference in New Issue
Block a user