From c66bf1eceba365623d6202e439f826915072ed6e Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Fri, 10 Jul 2026 16:25:01 +0200 Subject: [PATCH] fix(monitoring): pg_archive_stalled must not false-fire on an idle database MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "WAL archiving stalled" alert fired on prod during a quiet period. An idle Postgres archives nothing — it never force-switches an empty WAL segment on archive_timeout — so pg_stat_archiver_last_archive_age grows past 30 min even though archiving is perfectly healthy (failed_count 0, no .ready segments, pg_wal flat, backups current). A false positive: no data and no disk at risk (nothing was written, so the frozen recovery point equals the live state). Gate the age condition on pg_wal actually growing: (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB) so it fires only when WAL is being produced but not archived (the real "pg_wal fills the disk" danger; genuine archive_command failures are already caught by pg_archive_failing). `and on()` bridges the two metrics' differing label sets (last_archive_age carries a server label, the pg_wal gauge does not); delta() (not increase) suits the gauge. pg_stat_wal is not exported by this postgres_exporter, so pg_wal size growth is the available signal. Validated on prod with promtool: the expression is empty while idle. Also fix the runbook command in both archive alerts' annotations: `pgbackrest check` needs `--pg1-user=scrabble`, or it connects as role "root" (which does not exist) and aborts with "no database found". --- deploy/README.md | 6 ++++-- deploy/grafana/provisioning/alerting/rules.yaml | 14 ++++++++++---- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/deploy/README.md b/deploy/README.md index a10ece9..60e03af 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -255,8 +255,10 @@ shipping or redeploying this stack does **not** start archiving — the artifact armed, which is why an un-armed prod deploy can never pile WAL onto the disk. The base-backup timer is provisioned by the Ansible `main` role behind `pitr_enabled` (also default off). Two Grafana alerts watch health: `WAL archiving failing` (`pg_stat_archiver_failed_count` rising) -and `WAL archiving stalled` (`pg_stat_archiver_last_archive_age` over 30 min) — both -absent/NaN-safe, so they stay quiet until archiving is armed. +and `WAL archiving stalled` (last archive over 30 min old **while `pg_wal_size_bytes` is growing** +— the pg_wal-growth guard keeps an idle database, which archives nothing because it writes nothing, +from false-triggering during quiet hours) — both absent/NaN-safe, so they stay quiet until +archiving is armed. **Assessment (owner-reviewed; the gate before the first real money).** Measured on prod `pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At diff --git a/deploy/grafana/provisioning/alerting/rules.yaml b/deploy/grafana/provisioning/alerting/rules.yaml index c56aae6..9657c23 100644 --- a/deploy/grafana/provisioning/alerting/rules.yaml +++ b/deploy/grafana/provisioning/alerting/rules.yaml @@ -245,8 +245,14 @@ groups: conditions: - evaluator: { type: gt, params: [0] } labels: { severity: critical } - annotations: { summary: 'pgBackRest archive_command is failing — PITR is degrading and pg_wal can fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble check`.' } + annotations: { summary: 'pgBackRest archive_command is failing — PITR is degrading and pg_wal can fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check`.' } + # Fires only when the last archive is >30 min old AND pg_wal is actually growing (WAL is being + # produced but not archived). A genuinely idle database archives nothing — Postgres does not + # force-switch an empty segment on archive_timeout — so `last_archive_age` alone false-triggers + # during quiet hours; the pg_wal-growth guard removes that. `and on()` bridges the differing + # label sets (last_archive_age has a server label, the pg_wal gauge does not); delta() (not + # increase) suits the pg_wal_size_bytes gauge. Real archive failures are caught by pg_archive_failing. - uid: pg_archive_stalled title: WAL archiving stalled condition: C @@ -255,11 +261,11 @@ groups: execErrState: OK data: - refId: A - relativeTimeRange: { from: 300, to: 0 } + relativeTimeRange: { from: 2100, to: 0 } datasourceUid: prometheus model: refId: A - expr: pg_stat_archiver_last_archive_age + expr: (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16777216) instant: true - refId: C datasourceUid: __expr__ @@ -270,4 +276,4 @@ groups: conditions: - evaluator: { type: gt, params: [1800] } labels: { severity: critical } - annotations: { summary: 'No WAL segment archived for over 30 minutes (archive_timeout is 5m) — archiving is stalled; the PITR recovery point is frozen and pg_wal may grow.' } + annotations: { summary: 'No WAL archived for over 30 minutes while pg_wal keeps growing — archiving is genuinely stalled; the PITR recovery point is frozen and pg_wal will fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check`.' } -- 2.52.0