galaxy-game/notification/docs/runbook.md

# Operator Runbook

This runbook covers startup, steady-state verification, shutdown, and common
`Notification Service` incidents.

## Startup Checks

Before starting the process, confirm:

- `NOTIFICATION_REDIS_ADDR` points to the Redis deployment that stores
  notification records, routes, idempotency reservations, malformed intents,
  dead letters, stream offsets, and route schedules
- Redis ACL, DB, TLS, and timeout settings match the target environment
- `NOTIFICATION_USER_SERVICE_BASE_URL` points to the trusted internal
  `User Service`
- `NOTIFICATION_GATEWAY_CLIENT_EVENTS_STREAM` matches the stream consumed by
  `Gateway`
- `NOTIFICATION_MAIL_DELIVERY_COMMANDS_STREAM` matches the stream consumed by
  `Mail Service`
- administrator email variables are populated for notification types that
  should notify administrators
- OpenTelemetry exporter settings point at the intended collector when traces
  or metrics are expected outside the process

At startup the process performs a bounded Redis `PING`. Startup fails fast if
configuration validation or Redis connectivity fails.

Known startup caveats:

- there is no operator API
- there is no `/metrics` route
- traces and metrics are exported only through configured OpenTelemetry
  exporters
- readiness is process-local after successful startup

## Steady-State Verification

Practical readiness verification:

1. confirm startup logs for the internal HTTP listener, intent consumer, push
   publisher, and email publisher
2. request `GET /readyz` on `NOTIFICATION_INTERNAL_HTTP_ADDR`
3. verify Redis connectivity and OpenTelemetry exporter health out of band
4. publish a low-risk compatible test intent in a non-production environment
   and verify route publication in the downstream stream

Expected steady-state signals:

- `notification.route_schedule.depth` remains bounded
- `notification.route_schedule.oldest_age_ms` stays near the active retry
  ladder
- `notification.intent_stream.oldest_unprocessed_age_ms` remains near zero
  when producers are healthy
- `notification.route.dead_letters` changes rarely
- malformed-intent logs appear only for bad producer input
- logs include `notification_type`, `producer`, `audience_kind`, and
  correlation identifiers where present

## Shutdown

The process handles `SIGINT` and `SIGTERM`.

Shutdown behavior:

- coordinated shutdown is bounded by `NOTIFICATION_SHUTDOWN_TIMEOUT`
- the private probe listener is stopped before process resources are closed
- route publishers and the intent consumer stop through context cancellation
- Redis clients are closed after the app stops
- OpenTelemetry providers are flushed during runtime cleanup

During a planned restart:

1. send `SIGTERM`
2. wait for listener and worker shutdown logs
3. restart the process with the same Redis, stream, and downstream settings
4. repeat steady-state verification

## Incident Triage

### Intent Stream Lag Grows

Symptoms:

- `notification.intent_stream.oldest_unprocessed_age_ms` increases
- no matching route records appear for new stream entries
- consumer logs stop after a specific stream entry

Checks:

1. inspect the next unprocessed `notification:intents` entry
2. confirm `User Service` is reachable from `Notification Service`
3. if the entry is user-targeted, verify every `recipient_user_id` exists
4. inspect malformed-intent records for nearby stream IDs

Expected behavior:

- malformed input is recorded and the offset advances
- temporary `User Service` failure stops progress before offset advancement

### Route Schedule Backlog Grows

Symptoms:

- `notification.route_schedule.depth` rises steadily
- `notification.route_schedule.oldest_age_ms` increases
- routes remain in `pending` or `failed`

Checks:

1. confirm push and email publisher startup logs are present
2. confirm Redis latency and connectivity
3. verify route IDs match the expected `push:` or `email:` prefixes
4. confirm the downstream stream names match `Gateway` and `Mail Service`
5. inspect route `last_error_classification`

### Dead-Letter Spikes

Symptoms:

- `notification.route.dead_letters` increases rapidly
- route records show repeated `payload_encoding_failed`,
  `gateway_stream_publish_failed`, or `mail_stream_publish_failed`

Checks:

1. inspect the dead-letter entry and owning route
2. verify payload fields still match the notification catalog
3. confirm downstream Redis stream writes are accepted
4. compare failures across channels to isolate Gateway-specific or
   Mail-specific issues

Recovery:

1. correct the downstream dependency or payload problem
2. publish a new compatible intent with a new producer-owned
   `idempotency_key`
3. keep the old dead-letter record untouched as audit history

### Missing Administrator Mail

Symptoms:

- administrator notification type is accepted
- no email command reaches `mail:delivery_commands`
- route is `skipped` with recipient `config:<notification_type>`

Checks:

1. inspect the type-specific administrator email environment variable
2. confirm addresses are normalized single email addresses without display
   names
3. restart the process after configuration changes

Expected behavior:

- empty administrator lists materialize one skipped synthetic route so the
  configuration gap remains durable and visible

### Auth-Code Mail Appears Missing

Auth-code mail is intentionally outside `Notification Service`.

Checks:

1. inspect `Auth / Session Service -> Mail Service` logs and delivery records
2. confirm `notification:intents` remains unused for auth-code delivery
3. do not replay auth-code mail through `Notification Service`