galaxy-game/notification/docs/postgres-migration.md

PostgreSQL Migration

PG_PLAN.md §5 migrated galaxy/notification from a Redis-only durable store to the steady-state split codified in ARCHITECTURE.md §Persistence Backends: PostgreSQL is the source of truth for table-shaped notification state, and Redis keeps only the inbound notification:intents stream, the two outbound streams (gateway:client-events, mail:delivery_commands), the persisted consumer offset, and the short-lived per-route exclusivity lease.

This document records the schema decisions and the non-obvious agreements behind them. Use it together with the migration script (internal/adapters/postgres/migrations/00001_init.sql) and the runtime wiring (internal/app/runtime.go).

Outcomes

• Schema notification (provisioned externally) holds the durable state: records, routes, dead_letters, malformed_intents.
• The runtime opens one PostgreSQL pool via pkg/postgres.OpenPrimary, applies embedded goose migrations strictly before any HTTP listener becomes ready, and exits non-zero when migration or ping fails.
• The runtime opens one shared *redis.Client via pkg/redisconn.NewMasterClient and passes it to the intent consumer, the publishers (outbound XADDs), the route lease store, and the persisted stream offset store.
• The Redis adapter package (internal/adapters/redisstate/) is reduced to the surviving LeaseStore, StreamOffsetStore, and a slim Keyspace exposing only RouteLease(notificationID, routeID), StreamOffset(stream), and Intents(). The Lua-backed atomic writer, the route-state mutation scripts, the records/routes/idempotency/dead- letters/malformed-intents keyspace, and the per-record TTL constants are gone.
• Configuration drops NOTIFICATION_REDIS_USERNAME / NOTIFICATION_REDIS_TLS_ENABLED / NOTIFICATION_REDIS_ADDR and introduces NOTIFICATION_REDIS_MASTER_ADDR / NOTIFICATION_REDIS_REPLICA_ADDRS plus NOTIFICATION_POSTGRES_*. The retention knobs NOTIFICATION_RECORD_TTL / NOTIFICATION_DEAD_LETTER_TTL are renamed to NOTIFICATION_RECORD_RETENTION / NOTIFICATION_MALFORMED_INTENT_RETENTION, and a new NOTIFICATION_CLEANUP_INTERVAL drives the periodic SQL retention worker.

Decisions

1. One schema, externally-provisioned role

Decision. The notification schema and the matching notificationservice role are created outside the migration sequence (in tests, by integration/internal/harness/postgres_container.go::EnsureRoleAndSchema; in production, by an ops init script not in scope for this stage). The embedded migration 00001_init.sql only contains DDL for tables and indexes and assumes it runs as the schema owner with search_path=notification.

Why. Mixing role creation, schema creation, and table DDL into one script forces every consumer of the migration to run as a superuser. The schema-per-service architectural rule (ARCHITECTURE.md §Persistence Backends) lines up neatly with the operational split: ops provisions roles and schemas, the service applies schema-scoped migrations.

2. Idempotency record IS the records row

Decision. The records table carries producer, idempotency_key, request_fingerprint, and idempotency_expires_at columns and a UNIQUE (producer, idempotency_key) constraint. Acceptance flows insert the row directly; a duplicate request races on the UNIQUE constraint and surfaces as acceptintent.ErrConflict. There is no separate idempotency table.

Why. PG_PLAN.md §3 fixed this rule for every PG-backed service. With the reservation living on the durable record, recovery is a single fact — the row either exists or it does not — so no Redis-loss window can make a duplicate sneak through. The records.accepted_at value doubles as the IdempotencyRecord.CreatedAt returned to the service layer.

3. recipient_user_ids as JSONB

Decision. records.recipient_user_ids stores the normalized recipient user-id list as a JSONB column. The codec round-trips a nil slice as [] to keep the column NOT NULL while letting the read path return a nil slice when the audience is not user-targeted.

Why. The list is opaque to queries (we never element-filter on it). JSONB lines up with the "everything outside primary fields is JSON" pattern Mail Stage 4 already established; PostgreSQL will accept a future GIN index on recipient_user_ids jsonb_path_ops if a recipient-filtered operator UI ever lands. text[] would have forced a pgtype.Array[string] boundary type and a different scan path with no functional benefit today.

4. Timestamps are uniformly timestamptz and always UTC at the boundary

Decision. Every time-valued column on every Stage 5 table uses PostgreSQL's timestamptz. The domain model continues to use time.Time; the adapter normalises every time.Time parameter to UTC at the binding site (record.X.UTC() or the nullableTime helper that wraps a possibly zero-valued time.Time), and re-wraps every scanned time.Time with .UTC() (directly or via timeFromNullable for nullable columns) before it leaves the adapter. The architecture-wide form of this rule lives in ARCHITECTURE.md §Persistence Backends → Timestamp handling.

Why. PG_PLAN.md §5 originally specified _ms epoch-millisecond columns. User Service Stage 3 and Mail Service Stage 4 already use timestamptz for every table and the runtime contract tests expect Go-level time.Time semantics throughout. Keeping the same shape across services reduces adapter-layer complexity and avoids two parallel encoding paths in the notificationstore. The deviation from the literal plan is intentional and is documented here. The defensive .UTC() rule on both sides eliminates the class of bug where the pgx driver returns scanned values in time.Local, which silently breaks equality tests, JSON formatting, and comparison against pointer fields.

5. Scheduler claim is non-locking; transitions use optimistic concurrency on updated_at

Decision. ListDueRoutes(ctx, now, limit) is a non-locking SELECT notification_id, route_id FROM routes WHERE next_attempt_at IS NOT NULL AND next_attempt_at <= $1 ORDER BY next_attempt_at ASC LIMIT $2. The publisher then takes a Redis lease (route_leases:*), reads the route, emits the outbound stream entry, and calls one of CompleteRoutePublished / CompleteRouteFailed / CompleteRouteDeadLetter. Each Complete* transaction issues UPDATE routes SET ... WHERE notification_id = $a AND route_id = $b AND updated_at = $expectedUpdatedAt; a zero RowsAffected count surfaces as routestate.ErrConflict, which the publisher treats as a no-op (some other replica progressed the row since the worker loaded it).

Why. A FOR UPDATE held across the publisher's whole publish window would serialise concurrent publishers and block the outbound stream emit. Per-row optimistic concurrency on updated_at keeps the lock duration inside the SQL transaction itself; the lease bounds duplicates atop that. The explicit next_attempt_at column (set to NULL for terminal states) keeps the partial index routes_due_idx narrow and avoids the "schedule out of sync with row" failure mode of the previous Redis ZSET + JSON-payload pair.

6. Outbound XADD precedes SQL completion (at-least-once across the dual-system boundary)

Decision. The publisher emits the outbound stream entry through *redis.Client.XAdd before the route's SQL state transition is committed. If the XADD succeeds and the SQL update later fails, the next replica retries — same notification gets a second outbound entry; the consumer side (Gateway, Mail) deduplicates on the entry id. If the XADD fails, recordFailure records a publication failure with classification gateway_stream_publish_failed or mail_stream_publish_failed and schedules a retry.

Why. PG_PLAN.md §5 explicitly endorses this ordering by saying the lease is "atop the SQL claim" rather than replacing it. The lease bounds duplicate emission to one replica per route per lease window; the consumer-side dedupe handles the rare cross-window case. A transactional outbox would solve the duplicate but is out of Stage 5 scope; revisit if duplicate-traffic ever becomes an operational concern.

7. Lease stays on Redis as a hint

Decision. The lease key notification:route_leases:<notificationID>:<routeID> keeps its existing SETNX/Lua-release semantics, lifted into a dedicated redisstate.LeaseStore. The composite internal/adapters/postgres/routepublisher.Store wires the SQL state store and the Redis lease store behind the existing publisher-worker interfaces (PushRouteStateStore, EmailRouteStateStore).

Why. PG_PLAN.md §5 retains the lease as a "short-lived, per-process exclusivity hint atop the SQL claim". Without the lease, two replicas selecting overlapping due batches would each XADD before either commits the SQL transition — duplicating outbound traffic during contention. The lease bounds emission rate to one-per-route-per-lease-TTL even when scans overlap. Keeping the abstraction inside LeaseStore (separate from the SQL store) keeps the architectural split visible.

8. Periodic SQL retention replaces Redis EXPIRE

Decision. A new worker.SQLRetentionWorker runs the two DELETE statements driven by config:

• DELETE FROM records WHERE accepted_at < now() - $record_retention cascades to routes and dead_letters via ON DELETE CASCADE.
• DELETE FROM malformed_intents WHERE recorded_at < now() - $malformed_intent_retention is a standalone retention pass.

Three new env vars (NOTIFICATION_RECORD_RETENTION, NOTIFICATION_MALFORMED_INTENT_RETENTION, NOTIFICATION_CLEANUP_INTERVAL) drive the worker. NOTIFICATION_IDEMPOTENCY_TTL survives unchanged: the service layer materialises it on each row as idempotency_expires_at.

Why. PostgreSQL maintains its own indexes; the previous per-key Redis EXPIRE TTL semantics translate to a periodic batch DELETE. The two-knob shape mirrors Mail Stage 4 (MAIL_DELIVERY_RETENTION + MAIL_MALFORMED_COMMAND_RETENTION). The legacy NOTIFICATION_RECORD_TTL / NOTIFICATION_DEAD_LETTER_TTL env vars are intentionally retired without a backward-compat shim — keeping the names would mislead operators reading the runbook because the eviction mechanism genuinely changed.

9. Shared Redis client with consumer-driven shutdown

Decision. internal/app/runtime.go constructs one redisconn.NewMasterClient(cfg.Redis.Conn) (via the thin redisadapter.NewClient wrapper) and passes it to the intent consumer, the lease store, the stream offset store, and both publishers (for their outbound XADDs). The runtime cleanup tolerates redis.ErrClosed so a double-close from any consumer is benign.

Why. Each subsequent PG_PLAN stage (Lobby) ships a similar pattern; sharing one client is the shape we want all stages to converge on. A dedicated client per consumer is the artefact the Redis-only architecture needed; sharing one client multiplies fewer TCP connections, ping points, and OpenTelemetry instrumentation hooks for no functional benefit.

10. Query layer is go-jet/jet/v2

Decision. All notificationstore packages build SQL through the jet builder API (pgtable.<Table>.INSERT/SELECT/UPDATE/DELETE plus the pg.AND/OR/SET/MIN/COUNT/... DSL). cmd/jetgen (invoked via make jet) brings up a transient PostgreSQL container, applies the embedded migrations, and runs github.com/go-jet/jet/v2/generator/postgres.GenerateDB against the provisioned schema; the generated table/model code lives under internal/adapters/postgres/jet/notification/{model,table}/*.go and is committed to the repo, so build consumers do not need Docker. Statements are run through the database/sql API (stmt.Sql() → db/tx.Exec/Query/QueryRow); manual rowScanner helpers preserve the codecs.go boundary translations and domain-type mapping.

Why. Aligns with PG_PLAN.md §Library stack ("Query layer: github.com/go-jet/jet/v2 (PostgreSQL dialect). Generated code lives under each service internal/adapters/postgres/jet/, regenerated via a make jet target and committed to the repo"). Constructs the jet builder does not cover natively (MIN(timestamptz) aggregates, optimistic-concurrency WHERE updated_at = $expected, JSONB params) are expressed through the per-DSL helpers (pg.MIN(...), pg.TimestampzT(...), direct []byte/string params for JSONB columns).

Cross-References

• PG_PLAN.md §5 (Stage 5 — Notification Service migration).
• ARCHITECTURE.md §Persistence Backends.
• internal/adapters/postgres/migrations/00001_init.sql and internal/adapters/postgres/migrations/migrations.go.
• internal/adapters/postgres/notificationstore/{store,records,routes, acceptance,scheduler,dead_letters,malformed_intents,retention,codecs, helpers}.go plus the testcontainers-backed unit suite under notificationstore/{harness,store}_test.go.
• internal/adapters/postgres/jet/notification/{model,table}/*.go (committed generated code) plus cmd/jetgen/main.go and the make jet Makefile target that regenerate it.
• internal/adapters/postgres/routepublisher/store.go (composite PG state + Redis lease behind the publisher contracts).
• internal/service/routestate/types.go (storage-agnostic value types).
• internal/config/{config,env}.go (PostgresConfig plus the redisconn.Config-shaped RedisConfig envelope).
• internal/app/runtime.go (shared Redis client + PG pool open + migration
  • notificationstore wiring + retention worker startup).
• internal/worker/sqlretention.go (periodic SQL retention worker).
• internal/adapters/redisstate/{keyspace,codecs,errors,lease_store, stream_offset_store}.go (surviving slim Redis surface).
• integration/internal/harness/notificationservice.go (per-suite Postgres container + notification/notificationservice provisioning).