galaxy-game/mail/docs/postgres-migration.md

PostgreSQL Migration

PG_PLAN.md §4 migrated galaxy/mail from a Redis-only durable store to the steady-state split codified in ARCHITECTURE.md §Persistence Backends: PostgreSQL is the source of truth for table-shaped business state, and Redis keeps only the inbound mail:delivery_commands stream and its persisted consumer offset.

This document records the schema decisions and the non-obvious agreements behind them. Use it together with the migration script (internal/adapters/postgres/migrations/00001_init.sql) and the runtime wiring (internal/app/runtime.go).

Outcomes

• Schema mail (provisioned externally) holds the durable state: deliveries, delivery_recipients, attempts, dead_letters, delivery_payloads, malformed_commands.
• The runtime opens one PostgreSQL pool via pkg/postgres.OpenPrimary, applies embedded goose migrations strictly before any HTTP listener becomes ready, and exits non-zero when migration or ping fails.
• The runtime opens one shared *redis.Client via pkg/redisconn.NewMasterClient and passes it to the command consumer and the stream offset store; both stores no longer hold their own connection topology fields.
• The Redis adapter package (internal/adapters/redisstate/) is reduced to the surviving StreamOffsetStore plus a slim Keyspace exposing only StreamOffset(stream) and DeliveryCommands(). The Lua-backed atomic writer, the secondary index keys, the recipient/template/status indexes, the idempotency keyspace, and the per-record TTL constants are gone.
• Configuration drops MAIL_REDIS_USERNAME, MAIL_REDIS_TLS_ENABLED, MAIL_REDIS_ATTEMPT_SCHEDULE_KEY, MAIL_REDIS_DEAD_LETTER_PREFIX, MAIL_DELIVERY_TTL, and MAIL_ATTEMPT_TTL. MAIL_REDIS_ADDR becomes MAIL_REDIS_MASTER_ADDR + optional MAIL_REDIS_REPLICA_ADDRS. PostgreSQL-specific knobs live under MAIL_POSTGRES_*. New retention knobs (MAIL_DELIVERY_RETENTION, MAIL_MALFORMED_COMMAND_RETENTION, MAIL_CLEANUP_INTERVAL) drive a periodic SQL retention worker.

Decisions

1. One schema, externally-provisioned role

Decision. The mail schema and the matching mailservice role are created outside the migration sequence (in tests, by integration/internal/harness/postgres_container.go::EnsureRoleAndSchema; in production, by an ops init script not in scope for this stage). The embedded migration 00001_init.sql only contains DDL for tables and indexes and assumes it runs as the schema owner with search_path=mail.

Why. Mixing role creation, schema creation, and table DDL into one script forces every consumer of the migration to run as a superuser. The schema-per-service architectural rule (ARCHITECTURE.md §Persistence Backends) lines up neatly with the operational split: ops provisions roles and schemas, the service applies schema-scoped migrations.

2. Idempotency record IS the deliveries row

Decision. The deliveries table carries source, idempotency_key, request_fingerprint, and idempotency_expires_at columns and a UNIQUE (source, idempotency_key) constraint. Acceptance flows insert the row directly; a duplicate request races on the UNIQUE constraint and surfaces as acceptauthdelivery.ErrConflict / acceptgenericdelivery.ErrConflict. There is no separate idempotency table.

Why. PG_PLAN.md §3 fixed this rule for every PG-backed service. With the reservation living on the durable record, recovery is a single fact ("the row either exists or it does not"); no Redis-loss window can make a duplicate sneak through. Resend deliveries store an empty request_fingerprint and a synthetic far-future idempotency_expires_at; the read helper treats those rows as non-idempotent so future operator queries cannot mistake a clone for a hit.

3. Recipients live in a normalised side table

Decision. A delivery_recipients(delivery_id, kind, position, email) table stores envelope addresses with a kind CHECK constraint ('to'|'cc'|'bcc'|'reply_to') and an email index that excludes reply_to. The deliveries row does not embed envelope JSON.

Why. PG_PLAN.md §4 prescribed INDEX on … recipient as needed. A normalised table makes future recipient-filtered listing slot in without schema work and lets the existing operator listing implement the recipient filter as delivery_id IN (SELECT … FROM delivery_recipients WHERE … lower(email) = lower($1)). The Redis adapter previously maintained one index key per recipient — the same observable behaviour now comes for free from the PostgreSQL row layout plus a single index.

4. Timestamps are uniformly timestamptz and always UTC at the boundary

Decision. Every time-valued column on every Stage 4 table uses PostgreSQL's timestamptz. The domain model continues to use time.Time / *time.Time; the adapter normalises every time.Time parameter to UTC at the binding site (record.X.UTC() or the nullableTime helper that wraps *time.Time), and re-wraps every scanned time.Time with .UTC() (directly or via timeFromNullable) before it leaves the adapter. The architecture-wide form of this rule lives in ARCHITECTURE.md §Persistence Backends → Timestamp handling.

Why. PG_PLAN.md §4 originally specified mixed naming (timestamptz on deliveries, bigint epoch-ms on attempts/dead_letters/ malformed_commands). User Service Stage 3 already uses timestamptz for every table and the runtime contract tests expect Go-level time.Time semantics throughout. Keeping the same shape across services reduces adapter-layer complexity and avoids two parallel encoding paths in the mailstore. The deviation from the literal plan is intentional and is documented here. The defensive UTC rule on both sides eliminates the class of bug where the pgx driver returns scanned values in time.Local, which silently breaks equality tests, JSON formatting, and comparison against pointer fields.

5. Attempt scheduler reads via FOR UPDATE SKIP LOCKED

Decision. The attempt scheduler uses two indexed predicates:

• SELECT delivery_id FROM deliveries WHERE next_attempt_at IS NOT NULL AND next_attempt_at <= $now ORDER BY next_attempt_at ASC LIMIT $n to surface due deliveries (partial index deliveries_due_idx).
• SELECT … FROM deliveries WHERE delivery_id = $id AND status IN ('queued','rendered') AND next_attempt_at IS NOT NULL AND next_attempt_at <= $now FOR UPDATE SKIP LOCKED inside the claim transaction.

The next_attempt_at column is maintained explicitly: acceptance and attempt-commit transactions write it from the active scheduled attempt; claim sets it to NULL (the row is sending and the row stops being a scheduling candidate); a recovery commit re-populates it for the next attempt.

Why. FOR UPDATE SKIP LOCKED lets multiple scheduler instances run concurrently without serialising work on a single sorted set. Maintaining next_attempt_at in lockstep with the active attempt keeps the partial index small and avoids reading attempt rows during the hot-path schedule query. The previous Redis ZSET sort key was implicit; the SQL column is explicit, which removes a class of "the index is out of sync with the record" bugs that Lua-coordinated mutations made possible.

6. Recovery uses the most-recent attempt by exact attempt_no

Decision. LoadWorkItem(deliveryID) reads the delivery row and then the attempt row whose attempt_no = delivery.attempt_count. Concurrent commits that update the count and insert a new attempt are tolerated: the load lookup uses an exact key and never observes a partial state.

Why. A naive ORDER BY attempt_no DESC LIMIT 1 racing against a commit that already wrote the next attempt but had not yet committed the parent delivery row could observe attempt_no=count+1 while the delivery still reports count. Keying the read by the count deterministically returns the delivery's view of its own active attempt even under concurrent worker progress.

7. Periodic SQL retention replaces Redis index cleanup

Decision. A new worker.SQLRetentionWorker runs the two DELETE statements driven by config:

• DELETE FROM deliveries WHERE created_at < now() - $delivery_retention cascades to attempts, dead_letters, delivery_payloads, and delivery_recipients via ON DELETE CASCADE.
• DELETE FROM malformed_commands WHERE recorded_at < now() - $malformed_retention is a standalone retention pass.

Three new env vars (MAIL_DELIVERY_RETENTION, MAIL_MALFORMED_COMMAND_RETENTION, MAIL_CLEANUP_INTERVAL) drive the worker. MAIL_IDEMPOTENCY_TTL survives unchanged: it controls the per-acceptance idempotency_expires_at column the service layer materialises on each row.

Why. PostgreSQL maintains its own indexes; the previous redisstate.IndexCleaner had nothing to do once secondary index keys were gone. A per-table retention worker is the simplest model that keeps the mail database from accumulating audit history forever, while leaving the per-acceptance idempotency window controlled by its existing knob.

8. Shared Redis client with consumer-driven shutdown

Decision. internal/app/runtime.go constructs one redisconn.NewMasterClient(cfg.Redis.Conn) and passes it to both the stream offset store and the command consumer. The consumer's Shutdown closes the shared client to break the in-flight blocking XREAD; the runtime's cleanup function tolerates redis.ErrClosed so a double-close is benign.

Why. Each subsequent PG_PLAN stage (Notification, Lobby) ships a similar pattern; sharing one client is the shape we want all stages to converge on. The dedicated client for the consumer was an artefact of the Redis-only architecture and multiplied TCP connections, ping points, and OpenTelemetry instrumentation hooks for no functional benefit.

9. Query layer is go-jet/jet/v2

Decision. All mailstore packages build SQL through the jet builder API (pgtable.<Table>.INSERT/SELECT/UPDATE/DELETE plus the pg.AND/OR/SET/IN/... DSL). cmd/jetgen (invoked via make jet) brings up a transient PostgreSQL container, applies the embedded migrations, and runs github.com/go-jet/jet/v2/generator/postgres.GenerateDB against the provisioned schema; the generated table/model code lives under internal/adapters/postgres/jet/mail/{model,table}/*.go and is committed to the repo, so build consumers do not need Docker. Statements are run through the database/sql API (stmt.Sql() → db/tx.Exec/Query/QueryRow); manual scanners preserve the codecs.go boundary translations and domain-type mapping.

Why. Aligns with PG_PLAN.md §Library stack ("Query layer: github.com/go-jet/jet/v2 (PostgreSQL dialect). Generated code lives under each service internal/adapters/postgres/jet/, regenerated via a make jet target and committed to the repo"). Constructs the jet builder does not cover natively (FOR UPDATE, FOR UPDATE SKIP LOCKED, keyset-pagination row-comparison, JSONB params, LOWER(...) on subselects) are expressed through the per-DSL helpers (.FOR(pg.UPDATE()), .FOR(pg.UPDATE().SKIP_LOCKED()), pg.LOWER, OR/AND expansion of cursor predicates).

Cross-References

• PG_PLAN.md §4 (Stage 4 — Mail Service migration).
• ARCHITECTURE.md §Persistence Backends.
• internal/adapters/postgres/migrations/00001_init.sql and internal/adapters/postgres/migrations/migrations.go.
• internal/adapters/postgres/mailstore/{store,deliveries, auth_acceptance,generic_acceptance,render,operator, attempt_execution,malformed_command,codecs,helpers}.go plus the testcontainers-backed unit suite under mailstore/{harness,store}_test.go.
• internal/adapters/postgres/jet/mail/{model,table}/*.go (committed generated code) plus cmd/jetgen/main.go and the make jet Makefile target that regenerate it.
• internal/config/{config,env,validation}.go (PostgresConfig + the redisconn.Config-shaped Redis envelope).
• internal/app/{runtime,bootstrap}.go (shared Redis client + PG pool open + migration + mailstore wiring).
• internal/worker/sqlretention.go (periodic SQL retention worker).
• internal/adapters/redisstate/{keyspace,offset_codec,stream_offset_store}.go (surviving slim Redis surface).
• integration/internal/harness/mailservice.go (per-suite Postgres container + mail/mailservice provisioning).