# Operator Runbook

This runbook covers startup, steady-state verification, shutdown, and common
`Notification Service` incidents.

## Startup Checks

Before starting the process, confirm:

- `NOTIFICATION_REDIS_MASTER_ADDR` points to the Redis master deployment
  that hosts the inbound `notification:intents` stream, the persisted
  consumer offset, the outbound `gateway:client-events` and
  `mail:delivery_commands` streams, and the temporary `route_leases:*` keys
- `NOTIFICATION_REDIS_PASSWORD` matches the connection password
  (mandatory; the deprecated `NOTIFICATION_REDIS_USERNAME` /
  `NOTIFICATION_REDIS_TLS_ENABLED` env vars are rejected at startup)
- `NOTIFICATION_POSTGRES_PRIMARY_DSN` points to the PostgreSQL primary
  hosting the `notification` schema; the role must own
  `records`, `routes`, `dead_letters`, and `malformed_intents`
- `NOTIFICATION_USER_SERVICE_BASE_URL` points to the trusted internal
  `User Service`
- `NOTIFICATION_GATEWAY_CLIENT_EVENTS_STREAM` matches the stream consumed by
  `Gateway`
- `NOTIFICATION_MAIL_DELIVERY_COMMANDS_STREAM` matches the stream consumed by
  `Mail Service`
- administrator email variables are populated for notification types that
  should notify administrators
- retention knobs (`NOTIFICATION_RECORD_RETENTION`,
  `NOTIFICATION_MALFORMED_INTENT_RETENTION`,
  `NOTIFICATION_CLEANUP_INTERVAL`) are sized for the expected operator
  history window
- OpenTelemetry exporter settings point at the intended collector when traces
  or metrics are expected outside the process

At startup the process performs a bounded Redis `PING`, opens the
PostgreSQL pool, runs the embedded goose migrations, and only then starts
the internal HTTP probe. Startup fails fast if configuration validation,
Redis connectivity, PostgreSQL connectivity, or migration application
fails.

Known startup caveats:

- there is no operator API
- there is no `/metrics` route
- traces and metrics are exported only through configured OpenTelemetry
  exporters
- readiness is process-local after successful startup

## Steady-State Verification

Practical readiness verification:

1. confirm startup logs for the internal HTTP listener, intent consumer, push
   publisher, and email publisher
2. request `GET /readyz` on `NOTIFICATION_INTERNAL_HTTP_ADDR`
3. verify Redis connectivity and OpenTelemetry exporter health out of band
4. publish a low-risk compatible test intent in a non-production environment
   and verify route publication in the downstream stream

Expected steady-state signals:

- `notification.route_schedule.depth` remains bounded
- `notification.route_schedule.oldest_age_ms` stays near the active retry
  ladder
- `notification.intent_stream.oldest_unprocessed_age_ms` remains near zero
  when producers are healthy
- `notification.route.dead_letters` changes rarely
- malformed-intent logs appear only for bad producer input
- logs include `notification_type`, `producer`, `audience_kind`, and
  correlation identifiers where present

## Shutdown

The process handles `SIGINT` and `SIGTERM`.

Shutdown behavior:

- coordinated shutdown is bounded by `NOTIFICATION_SHUTDOWN_TIMEOUT`
- the private probe listener is stopped before process resources are closed
- route publishers and the intent consumer stop through context cancellation
- Redis clients are closed after the app stops
- OpenTelemetry providers are flushed during runtime cleanup

During a planned restart:

1. send `SIGTERM`
2. wait for listener and worker shutdown logs
3. restart the process with the same Redis, stream, and downstream settings
4. repeat steady-state verification

## Incident Triage

### Intent Stream Lag Grows

Symptoms:

- `notification.intent_stream.oldest_unprocessed_age_ms` increases
- no matching route records appear for new stream entries
- consumer logs stop after a specific stream entry

Checks:

1. inspect the next unprocessed `notification:intents` entry
2. confirm `User Service` is reachable from `Notification Service`
3. if the entry is user-targeted, verify every `recipient_user_id` exists
4. inspect malformed-intent records for nearby stream IDs

Expected behavior:

- malformed input is recorded and the offset advances
- temporary `User Service` failure stops progress before offset advancement

### Route Schedule Backlog Grows

Symptoms:

- `notification.route_schedule.depth` rises steadily
- `notification.route_schedule.oldest_age_ms` increases
- routes remain in `pending` or `failed`

Checks:

1. confirm push and email publisher startup logs are present
2. confirm Redis latency and connectivity
3. verify route IDs match the expected `push:` or `email:` prefixes
4. confirm the downstream stream names match `Gateway` and `Mail Service`
5. inspect route `last_error_classification`

### Dead-Letter Spikes

Symptoms:

- `notification.route.dead_letters` increases rapidly
- route records show repeated `payload_encoding_failed`,
  `gateway_stream_publish_failed`, or `mail_stream_publish_failed`

Checks:

1. inspect the dead-letter entry and owning route
2. verify payload fields still match the notification catalog
3. confirm downstream Redis stream writes are accepted
4. compare failures across channels to isolate Gateway-specific or
   Mail-specific issues

Recovery:

1. correct the downstream dependency or payload problem
2. publish a new compatible intent with a new producer-owned
   `idempotency_key`
3. keep the old dead-letter record untouched as audit history

### Missing Administrator Mail

Symptoms:

- administrator notification type is accepted
- no email command reaches `mail:delivery_commands`
- route is `skipped` with recipient `config:<notification_type>`

Checks:

1. inspect the type-specific administrator email environment variable
2. confirm addresses are normalized single email addresses without display
   names
3. restart the process after configuration changes

Expected behavior:

- empty administrator lists materialize one skipped synthetic route so the
  configuration gap remains durable and visible

### Auth-Code Mail Appears Missing

Auth-code mail is intentionally outside `Notification Service`.

Checks:

1. inspect `Auth / Session Service -> Mail Service` logs and delivery records
2. confirm `notification:intents` remains unused for auth-code delivery
3. do not replay auth-code mail through `Notification Service`