galaxy-game/mail/docs/runbook.md

# Operator Runbook

This runbook covers the checks that matter most during startup, steady-state
verification, shutdown, and common `Mail Service` incidents.

## Startup Checks

Before starting the process, confirm:

- `MAIL_REDIS_MASTER_ADDR` and `MAIL_REDIS_PASSWORD` point to the Redis
  deployment that hosts the inbound `mail:delivery_commands` Stream and the
  persisted consumer offset
- `MAIL_POSTGRES_PRIMARY_DSN` points to the PostgreSQL deployment whose
  `mail` schema (provisioned externally for the `mailservice` role) holds the
  durable mail state — deliveries, attempts, dead letters, payloads,
  idempotency reservations, malformed commands
- `MAIL_TEMPLATE_DIR` points to the intended immutable template catalog
- if `MAIL_SMTP_MODE=smtp`, the SMTP address, sender identity, and optional
  credentials are configured together
- the OpenTelemetry exporter settings point at the intended collector when
  traces or metrics are expected outside the process

At startup the process pings the shared Redis master client, opens the
PostgreSQL pool, applies embedded goose migrations strictly before any HTTP
listener opens, parses the full template catalog, and only then starts the
internal HTTP listener and background workers.

Startup fails fast if any of those steps fail.

Known startup caveats:

- there is no `/healthz`, `/readyz`, or `/metrics` route
- traces and metrics are exported only through the configured OpenTelemetry
  exporters
- template changes are not hot-reloaded; restart is required after template
  edits

## Steady-State Verification

Practical readiness verification is:

1. confirm the process emitted startup logs for the internal HTTP listener,
   command consumer, scheduler, attempt worker pool, and SQL retention
   worker
2. open a TCP connection to `MAIL_INTERNAL_HTTP_ADDR`
3. issue one trusted smoke request such as
   `GET /api/v1/internal/deliveries/does-not-exist`
4. verify Redis and PostgreSQL connectivity, plus OpenTelemetry exporter
   health, out of band

Expected steady-state signals:

- `mail.attempt_schedule.depth` remains bounded
- `mail.attempt_schedule.oldest_age_ms` stays near the active retry ladder
- `mail.delivery.dead_letters` changes rarely
- `mail.stream_commands.malformed` changes only on bad upstream commands
- internal HTTP logs include `otel_trace_id` and `otel_span_id`

## Shutdown

The process handles `SIGINT` and `SIGTERM`.

Shutdown behavior:

- coordinated shutdown is bounded by `MAIL_SHUTDOWN_TIMEOUT`
- the internal HTTP listener is stopped before process resources are closed
- the Redis master client and PostgreSQL pool are closed after the app stops
- OpenTelemetry providers are flushed during runtime cleanup

During a planned restart:

1. send `SIGTERM`
2. wait for listener and worker shutdown logs
3. restart the process with the same Redis, PostgreSQL, and template
   configuration
4. repeat the steady-state verification steps

## Incident Triage

### Attempt Schedule Backlog Grows

Symptoms:

- `mail.attempt_schedule.depth` rises steadily
- `mail.attempt_schedule.oldest_age_ms` increases instead of oscillating
- queued deliveries remain in `queued` or `rendered` longer than expected

Checks:

1. confirm the scheduler is still logging regular activity
2. confirm PostgreSQL connectivity and latency on the `deliveries`
   `(next_attempt_at)` partial index — scheduler claims rely on
   `FOR UPDATE SKIP LOCKED`, so contention here surfaces as backlog
3. confirm attempt workers are running and not blocked on SMTP
4. inspect `mail.provider.send.duration_ms` for elevated latency
5. verify `MAIL_ATTEMPT_WORKER_CONCURRENCY` is appropriate for the workload

### Dead-Letter Spikes

Symptoms:

- `mail.delivery.dead_letters` increases rapidly
- operator reads show repeated `dead_letter` deliveries with recent
  `transport_failed` or `timed_out` attempts

Checks:

1. inspect recent provider summaries on dead-lettered deliveries
2. confirm SMTP reachability from the Mail Service process
3. compare the spike against `mail.provider.send.duration_ms` and timeout logs
4. verify the remote SMTP server is accepting `STARTTLS` and mail submission

Expected behavior:

- dead letters appear only after the fixed retry ladder is exhausted
- each dead-lettered delivery has a matching dead-letter entry

### Repeated `suppressed` Outcomes

Symptoms:

- `mail.delivery.suppressed` rises unexpectedly
- auth or generic deliveries end as `suppressed`

Checks:

1. determine whether the source is `authsession` or `notification`
2. for auth deliveries, confirm the service is not intentionally running in
   `MAIL_SMTP_MODE=stub`
3. inspect provider summaries for policy-driven suppression markers
4. confirm the upstream business workflow still expects those deliveries to be
   skipped

Expected behavior:

- auth suppression is valid in stub mode and still counts as successful intake
- provider-side suppression is recorded as
  `mail_attempt.status=provider_rejected` together with
  `mail_delivery.status=suppressed`

### SMTP Authentication Failures

Symptoms:

- provider summaries indicate auth or login failures
- delivery attempts shift toward `failed` or repeated retryable failures,
  depending on provider classification

Checks:

1. verify `MAIL_SMTP_USERNAME` and `MAIL_SMTP_PASSWORD` are both configured
2. verify the credential pair is valid for the target SMTP server
3. verify the sender identity matches the allowed submission account
4. confirm the server advertises the expected authentication mechanisms

### SMTP Timeouts

Symptoms:

- `mail.attempt.outcomes{status="timed_out"}` increases
- `mail.provider.send.duration_ms` shifts upward
- logs show retry scheduling or dead-letter transitions after timeout paths

Checks:

1. confirm network reachability to `MAIL_SMTP_ADDR`
2. compare observed send duration with `MAIL_SMTP_TIMEOUT`
3. verify the SMTP server is not stalling during `STARTTLS`, auth, or `DATA`
4. confirm the process is not CPU-starved or blocked on Redis

### Malformed Stream Commands

Symptoms:

- `mail.stream_commands.malformed` increases
- logs contain `stream command rejected`

Checks:

1. inspect `failure_code`, `delivery_id`, `source`, and `stream_entry_id`
2. confirm the upstream command payload still matches
   [`../api/delivery-commands-asyncapi.yaml`](../api/delivery-commands-asyncapi.yaml)
3. confirm the producer still sends canonical `payload_mode`, locale, and
   idempotency fields
4. review stored malformed-command records through the operator tooling or
   direct Redis inspection