galaxy-game/backend/docs/runbook.md

# Operator Runbook

Practical pointers for operating `galaxy/backend` and the integration
test stack. The list mirrors the steady-state behaviour documented in
`../README.md`; when in doubt, the README is canonical.

## Cold start

1. Provision Postgres and configure `BACKEND_POSTGRES_DSN` with
   `?search_path=backend`.
2. Provision an SMTP relay reachable from the backend host. Use
   `BACKEND_SMTP_TLS_MODE=none` only for local development.
3. Mount a GeoLite2 Country `.mmdb` and point
   `BACKEND_GEOIP_DB_PATH` at it. The `pkg/geoip/test-data/` submodule
   ships a fixture that is sufficient for synthetic IPs.
4. Mount the Docker daemon socket if the deployment is responsible
   for engine containers. The MVP topology mounts
   `/var/run/docker.sock` directly; future hardening introduces a
   `tecnativa/docker-socket-proxy` sidecar.
5. Ensure the user-defined Docker bridge named in
   `BACKEND_DOCKER_NETWORK` exists; backend's
   `dockerclient.EnsureNetwork` creates it if missing on first boot.
6. Seed the bootstrap admin via `BACKEND_ADMIN_BOOTSTRAP_USER` and
   `BACKEND_ADMIN_BOOTSTRAP_PASSWORD`; rotate the password immediately
   after the first deploy through the admin surface. The insert is
   idempotent.

## Migrations

`pressly/goose/v3` applies embedded migrations from
`internal/postgres/migrations/`. The pre-production set ships as
`00001_init.sql` plus additive numbered files. Backend always runs
`CREATE SCHEMA IF NOT EXISTS backend` before goose so a fresh database
does not trip the bookkeeping table on the first migration.

`internal/postgres/migrations_test.go` asserts that the migration
produces the expected table set; adding a table without updating the
expected list is a loud test failure.

## Probes

- `GET /healthz` — process liveness. Always `200` once the binary is
  alive.
- `GET /readyz` — `200` once Postgres is reachable, migrations are
  applied, every cache warm-up has finished, and the gRPC push
  listener is bound. Returns `503` until all hold.

## Caches

Every cache (`auth`, `user`, `admin`, `lobby`, `runtime`,
`engineversion`) reads its full table at startup. Mutations write
through the cache *after* the matching Postgres mutation commits, so
a commit failure leaves the cache in sync with the previous database
state. To force a cache rebuild, restart the process; there is no
runtime invalidation endpoint.

## Mail outbox

- The worker scans every `BACKEND_MAIL_WORKER_INTERVAL` (default
  `2s`) using `SELECT ... FOR UPDATE SKIP LOCKED`.
- A row reaches `dead_lettered` after `BACKEND_MAIL_MAX_ATTEMPTS`
  (default `8`).
- Operators inspect the outbox via:
  - `GET /api/v1/admin/mail/deliveries?page=N`
  - `GET /api/v1/admin/mail/deliveries/{delivery_id}`
  - `GET /api/v1/admin/mail/deliveries/{delivery_id}/attempts`
  - `GET /api/v1/admin/mail/dead-letters`
- `POST /api/v1/admin/mail/deliveries/{delivery_id}/resend` re-arms a
  delivery for another attempt cycle. Allowed states are `pending`,
  `retrying`, and `dead_lettered`. Resend on a `sent` row returns
  `409 Conflict`.
- `mail_attempts.attempt_no` is monotonic across the entire history
  of a single delivery; a resend appends new attempts rather than
  starting over.

## Notification pipeline

- `notification.Submit(intent)` validates the intent shape, enforces
  idempotency via `UNIQUE (kind, idempotency_key)`, and materialises
  per-route rows in `notification_routes`. Push routes go straight to
  `push.Service`; email routes are inserted into `mail_deliveries`.
- The notification worker mirrors the mail worker pattern: `SELECT
  ... FOR UPDATE SKIP LOCKED` on `notification_routes`, scan every
  `BACKEND_NOTIFICATION_WORKER_INTERVAL` (default `5s`), dead-letter
  after `BACKEND_NOTIFICATION_MAX_ATTEMPTS` (default `8`).
- `OnUserDeleted` skips a user's pending routes rather than deleting
  them so audit trails are preserved.
- Admin-channel kinds (`runtime.image_pull_failed`,
  `runtime.container_start_failed`, `runtime.start_config_invalid`)
  deliver email to `BACKEND_NOTIFICATION_ADMIN_EMAIL`. When that
  variable is empty, routes land with `status='skipped'` so the
  catalog never silently discards an admin-targeted intent.

## Runtime control plane

- `runtime_operation_log` records every container operation (start,
  stop, patch, force-next-turn) with start/finish timestamps,
  outcome, and error message.
- `BACKEND_RUNTIME_RECONCILE_INTERVAL` (default `60s`) governs the
  reconciler. It walks `docker ps -f label=galaxy.backend=1` and
  reconciles against `runtime_records`.
- `BACKEND_RUNTIME_IMAGE_PULL_POLICY` accepts `if_missing` (default),
  `always`, `never`. `never` requires that the engine image be
  pre-pulled on every host that may run a game.
- Force-next-turn flips a one-shot skip flag in `runtime_records`;
  the next scheduled tick observes the flag and consumes it.

## Geo

- `accounts.declared_country` is set once at registration. There is
  no version history; admins inspect the current value through the
  user surface.
- `user_country_counters` is updated fire-and-forget per
  authenticated request. Lookups are best-effort: any `pkg/geoip`
  error is logged and ignored, never blocks the request.
- Source IP for both flows reads the leftmost `X-Forwarded-For` and
  falls back to `RemoteAddr`. Backend trusts the value because the
  trust boundary lives at gateway.
- Email PII never appears in logs verbatim. Modules emit a per-process
  HMAC-SHA256-truncated `email_hash` instead.

## Telemetry

- `BACKEND_OTEL_TRACES_EXPORTER` and
  `BACKEND_OTEL_METRICS_EXPORTER` accept `otlp` (default), `none`,
  `stdout`, and (metrics only) `prometheus`. The Prometheus path
  binds a separate listener at
  `BACKEND_OTEL_PROMETHEUS_LISTEN_ADDR` so the scrape endpoint stays
  off the public surface.
- Logs are JSON to stdout; crash dumps to stderr.
- `otel_trace_id` and `otel_span_id` are injected into every log line
  written inside a request scope, so a single `request_id` correlates
  across HTTP, gRPC, and the workers.

## Integration test suite

`integration/` boots the full stack (Postgres, Redis, mailpit,
backend, gateway, optionally a `galaxy-game` engine) through
`testcontainers-go`. Day-to-day commands:

```bash
# Run every scenario; first cold run builds the three Docker images.
go test ./integration/...

# Run a single scenario.
go test -count=1 -v -run TestAuthFlow ./integration/...

# Force a rebuild of the integration images.
docker rmi galaxy/backend:integration galaxy/gateway:integration galaxy/game:integration
go test ./integration/...
```

Each scenario calls `testenv.Bootstrap(t)` which spins up an isolated
stack and registers `t.Cleanup` for every container. On test failure,
backend and gateway container logs are dumped through `t.Logf`. The
backend container runs as uid 0 so it can read the Docker daemon
socket; production deployments run distroless `nonroot` and rely on a
docker-socket-proxy sidecar.

The integration suite is the only place that exercises the engine
container lifecycle end-to-end. Building `galaxy/game:integration`
adds ~30–60 seconds to a cold run; subsequent runs reuse the
BuildKit layer cache.