feat: use postgres

mail/README.md

@@ -50,13 +50,21 @@ Cross-service routing rules:
 `cmd/mail` starts one internal-only process with:
 
 - one trusted internal HTTP listener on `MAIL_INTERNAL_HTTP_ADDR`
-- one async command consumer
-- one attempt scheduler
+- one async command consumer reading from `MAIL_REDIS_COMMAND_STREAM`
+- one attempt scheduler driven by Postgres `FOR UPDATE SKIP LOCKED`
 - one attempt worker pool
-- one cleanup worker
+- one SQL retention worker
 
 The service has no public ingress and no dedicated admin listener.
 
+Persistence split (steady state, see `docs/postgres-migration.md`):
+
+- PostgreSQL is the source of truth for durable mail state — accepted
+  deliveries, attempts, dead letters, payload bundles, malformed-command
+  audit records, and idempotency reservations.
+- Redis is the source of truth only for the inbound `mail:delivery_commands`
+  stream and its persisted consumer offset.
+
 Intentional runtime omissions:
 
 - no `/healthz`
@@ -65,8 +73,10 @@ Intentional runtime omissions:
 
 Operational behavior:
 
-- startup performs bounded Redis connectivity checks and fails fast on invalid
-  runtime configuration
+- startup performs bounded Redis and PostgreSQL connectivity checks and fails
+  fast on invalid runtime configuration
+- embedded goose migrations are applied strictly before any HTTP listener
+  opens; a migration failure exits with non-zero status
 - the template catalog is parsed once at startup and kept immutable for the
   lifetime of the process
 - template changes require process restart
@@ -76,7 +86,9 @@ Operational behavior:
 
 Required for all starts:
 
-- `MAIL_REDIS_ADDR`
+- `MAIL_REDIS_MASTER_ADDR`
+- `MAIL_REDIS_PASSWORD`
+- `MAIL_POSTGRES_PRIMARY_DSN`
 
 Primary configuration groups:
 
@@ -88,13 +100,21 @@ Primary configuration groups:
   - `MAIL_INTERNAL_HTTP_READ_HEADER_TIMEOUT`
   - `MAIL_INTERNAL_HTTP_READ_TIMEOUT`
   - `MAIL_INTERNAL_HTTP_IDLE_TIMEOUT`
-- Redis connectivity:
-  - `MAIL_REDIS_USERNAME`
+- Redis connectivity (`pkg/redisconn` shape):
+  - `MAIL_REDIS_MASTER_ADDR`
+  - `MAIL_REDIS_REPLICA_ADDRS` (comma-separated, optional)
   - `MAIL_REDIS_PASSWORD`
   - `MAIL_REDIS_DB`
-  - `MAIL_REDIS_TLS_ENABLED`
   - `MAIL_REDIS_OPERATION_TIMEOUT`
   - `MAIL_REDIS_COMMAND_STREAM`
+- PostgreSQL connectivity (`pkg/postgres` shape):
+  - `MAIL_POSTGRES_PRIMARY_DSN`
+  - `MAIL_POSTGRES_REPLICA_DSNS` (comma-separated, optional; reserved for
+    future read routing)
+  - `MAIL_POSTGRES_OPERATION_TIMEOUT`
+  - `MAIL_POSTGRES_MAX_OPEN_CONNS`
+  - `MAIL_POSTGRES_MAX_IDLE_CONNS`
+  - `MAIL_POSTGRES_CONN_MAX_LIFETIME`
 - SMTP provider:
   - `MAIL_SMTP_MODE=stub|smtp`
   - `MAIL_SMTP_ADDR`
@@ -110,6 +130,11 @@ Primary configuration groups:
   - `MAIL_ATTEMPT_WORKER_CONCURRENCY`
   - `MAIL_STREAM_BLOCK_TIMEOUT`
   - `MAIL_OPERATOR_REQUEST_TIMEOUT`
+  - `MAIL_IDEMPOTENCY_TTL`
+- SQL retention worker:
+  - `MAIL_DELIVERY_RETENTION` (default `30d`)
+  - `MAIL_MALFORMED_COMMAND_RETENTION` (default `90d`)
+  - `MAIL_CLEANUP_INTERVAL` (default `1h`)
 - OpenTelemetry:
   - `OTEL_SERVICE_NAME`
   - `OTEL_TRACES_EXPORTER`
@@ -125,26 +150,27 @@ Defaults worth knowing:
 - `MAIL_INTERNAL_HTTP_ADDR=:8080`
 - `MAIL_SMTP_MODE=stub`
 - `MAIL_SMTP_TIMEOUT=15s`
-
-Additional SMTP note:
-
-- `MAIL_SMTP_INSECURE_SKIP_VERIFY=false` by default and is intended only for
-  local self-signed SMTP capture or similar non-production environments
 - `MAIL_TEMPLATE_DIR=templates`
 - `MAIL_ATTEMPT_WORKER_CONCURRENCY=4`
 - `MAIL_STREAM_BLOCK_TIMEOUT=2s`
 - `MAIL_OPERATOR_REQUEST_TIMEOUT=5s`
 - `MAIL_SHUTDOWN_TIMEOUT=5s`
+- `MAIL_IDEMPOTENCY_TTL=168h` (`7d`)
+- `MAIL_DELIVERY_RETENTION=720h` (`30d`)
+- `MAIL_MALFORMED_COMMAND_RETENTION=2160h` (`90d`)
+- `MAIL_CLEANUP_INTERVAL=1h`
 
-Current implementation caveats:
+Additional SMTP note:
 
-- `MAIL_REDIS_COMMAND_STREAM` is effective for the async command consumer
-- `MAIL_REDIS_ATTEMPT_SCHEDULE_KEY` and `MAIL_REDIS_DEAD_LETTER_PREFIX` are
-  parsed but the Redis adapters still use the fixed keys
-  `mail:attempt_schedule` and `mail:dead_letters:<delivery_id>`
-- `MAIL_IDEMPOTENCY_TTL`, `MAIL_DELIVERY_TTL`, and `MAIL_ATTEMPT_TTL` are
-  parsed but the Redis adapters still enforce fixed retentions of `7d`, `30d`,
-  and `90d`
+- `MAIL_SMTP_INSECURE_SKIP_VERIFY=false` by default and is intended only for
+  local self-signed SMTP capture or similar non-production environments
+
+Retired (Stage 4 of `PG_PLAN.md`): `MAIL_REDIS_ADDR`, `MAIL_REDIS_USERNAME`,
+`MAIL_REDIS_TLS_ENABLED`, `MAIL_REDIS_ATTEMPT_SCHEDULE_KEY`,
+`MAIL_REDIS_DEAD_LETTER_PREFIX`, `MAIL_DELIVERY_TTL`, `MAIL_ATTEMPT_TTL`.
+The new connection envelope is supplied by `pkg/redisconn` and `pkg/postgres`,
+and durable retention is enforced by the SQL retention worker against the
+PostgreSQL-backed source of truth (see `docs/postgres-migration.md`).
 
 ## Stable Input Contracts
 
@@ -370,47 +396,48 @@ Rendering rules:
 - missing required variables and template lookup failures are classified into
   stable render-failure codes
 
-## Redis Logical Model
+## Persistence Layout
 
-Primary keys:
+PostgreSQL `mail` schema (source of truth — see
+[`docs/postgres-migration.md`](docs/postgres-migration.md)):
 
-- `mail:deliveries:<delivery_id>`
-- `mail:attempts:<delivery_id>:<attempt_no>`
-- `mail:idempotency:<source>:<idempotency_key>`
-- `mail:dead_letters:<delivery_id>`
-- `mail:delivery_payloads:<delivery_id>`
-- `mail:malformed_commands:<stream_entry_id>`
-- `mail:stream_offsets:<stream>`
+- `deliveries(delivery_id PK, source, status, payload_mode, …,
+  idempotency_key, request_fingerprint, idempotency_expires_at,
+  attempt_count, next_attempt_at, created_at, updated_at, …)` with
+  `UNIQUE (source, idempotency_key)` and a partial scheduler index on
+  `next_attempt_at`
+- `delivery_recipients(delivery_id FK, kind, position, email)` with
+  `kind ∈ {'to','cc','bcc','reply_to'}` and an `email` index that excludes
+  `reply_to`
+- `attempts(delivery_id FK, attempt_no, status, scheduled_for, started_at,
+  finished_at, provider_classification, provider_summary)`,
+  `PRIMARY KEY (delivery_id, attempt_no)`
+- `dead_letters(delivery_id PK FK, final_attempt_no, failure_classification,
+  provider_summary, recovery_hint, created_at)`
+- `delivery_payloads(delivery_id PK FK, payload jsonb)` for raw attachment
+  bundles
+- `malformed_commands(stream_entry_id PK, delivery_id, source,
+  idempotency_key, failure_code, failure_message, raw_fields jsonb,
+  recorded_at)`
 
-Scheduling and ingress keys:
+Redis surface (intake stream + offset only):
 
-- `mail:delivery_commands`
-- `mail:attempt_schedule`
-
-Operator indexes:
-
-- `mail:idx:recipient:<email>`
-- `mail:idx:status:<status>`
-- `mail:idx:source:<source>`
-- `mail:idx:template:<template_id>`
-- `mail:idx:idempotency:<source>:<idempotency_key>`
-- `mail:idx:created_at`
-- `mail:idx:malformed_command:created_at`
+- `mail:delivery_commands` — async ingress Redis Stream
+- `mail:stream_offsets:<stream>` — persisted consumer offset for the
+  intake stream
 
 Storage rules:
 
-- dynamic Redis key segments are base64url-encoded
-- durable records are stored as strict JSON blobs
-- timestamps are stored in Unix milliseconds
-- raw attachment payloads are separated from audit metadata
+- timestamps are stored as PostgreSQL `timestamptz` and normalised to UTC
+  at the adapter boundary
 - malformed async commands are stored idempotently by `stream_entry_id`
-
-Current fixed retentions:
-
-- idempotency: `7d`
-- deliveries and payload audit: `30d`
-- attempts and dead letters: `90d`
-- malformed commands: `90d`
+- the `idempotency_expires_at` column is set per acceptance from
+  `MAIL_IDEMPOTENCY_TTL` (default `7d`); resends store an empty fingerprint
+  and a synthetic far-future expiry that the read helper treats as
+  non-idempotent
+- the SQL retention worker periodically deletes deliveries older than
+  `MAIL_DELIVERY_RETENTION` (cascade) and malformed commands older than
+  `MAIL_MALFORMED_COMMAND_RETENTION`
 
 ## Provider, Retry, and Failure Policy