# Testing

Test strategy and runbook for the [Galaxy Game](ARCHITECTURE.md)
platform. The platform ships three executables — `gateway`,
`backend`, `game` (the engine container) — plus the shared `pkg/*`
libraries. This document defines the layering of tests, the
mandatory minimum coverage per executable, the integration runbook,
and the principles every test must follow.

## Layers

1. **Service tests** verify a single executable in isolation. They
   live next to the implementation as `*_test.go` files and use only
   in-process or testcontainers-managed dependencies. The package
   either runs entirely in process or boots a single Postgres
   testcontainer per test.
2. **Inter-service integration tests** verify one cross-process seam
   between two real executables (most often `gateway ↔ backend`,
   sometimes `backend ↔ game`). They live in
   [`galaxy/integration/`](../integration/) and drive the platform
   from outside the trust boundary.
3. **Full system tests** are a small, focused subset of the
   integration suite that walks an entire user-facing flow from the
   client edge through every component the flow touches. They live
   in the same `integration/` module and reuse the same fixtures.

Service tests are the cheapest and the broadest; integration tests
are slower and broader; full-system tests are the slowest and the
narrowest. The pyramid stays in this order — never replace a service
test with a system test.

## Global rules

- Every executable owns the service tests for its packages. Adding a
  new package without `_test.go` files is a review block.
- Every cross-process seam must have at least one passing
  inter-service test before the seam is wired in production.
- Async flows (mail outbox, notification routes, runtime workers,
  push gRPC) get tests for both the success path and the retry /
  dead-letter path, and a duplicate-event safety check.
- Sync flows get happy path, validation failure, timeout
  propagation, and dependency unavailable.
- Every external or trusted-internal API must have contract tests
  alongside behaviour tests. `backend/internal/server/contract_test.go`
  is the reference; gateway runs the same shape against
  `gateway/openapi.yaml`.
- The integration suite must keep running on a developer machine
  with Docker available. The only acceptable `t.Skip` is
  `testenv.RequireDocker` (no daemon at all). Any failure deeper
  than that — `tcpostgres.Run`, network create, image build, schema
  migration — fails the test loudly with `t.Fatal`. The historical
  bug we fixed (silent skips on reaper failures masking 27
  integration tests as "ok") came from treating an environment
  break as a skip.

## Service-specific coverage

### `galaxy/gateway`

Service tests live under `gateway/internal/`:

- Public REST routing, error projection, and OpenAPI contract
  validation.
- Authenticated gRPC envelope verification (`grpcapi.Server`):
  signature, payload hash, freshness window, anti-replay reservation,
  unknown / revoked sessions.
- Session cache (`session.BackendCache`) — the only implementation
  in the codebase, a thin wrapper around the `backendclient.RESTClient`
  per-request lookup.
- Response signing for unary responses and stream events
  (`authn.ResponseSigner`).
- Push hub (`push.Hub`) and push fan-out (`push_fanout.go`).
- Replay store (`replay.RedisStore`) reservation semantics.
- Anti-abuse rate limits per IP / session / user / message class.

### `galaxy/backend`

Service tests live under `backend/internal/`:

- Startup wiring: `app.App` lifecycle, telemetry runtime, Postgres
  pool, embedded migrations.
- OpenAPI contract test (`internal/server/contract_test.go`):
  validates every documented operation against the live gin engine.
- Domain unit + e2e tests per package (`auth`, `user`, `admin`,
  `lobby`, `runtime`, `mail`, `notification`, `geo`, `push`).
  E2E tests (`*_e2e_test.go`) spin up a Postgres testcontainer.
- Mail outbox: pickup with `SELECT FOR UPDATE SKIP LOCKED`, retry
  with backoff plus jitter, dead-letter past `MAX_ATTEMPTS`,
  resend semantics (`pending|retrying|dead_lettered` → re-armed,
  `sent` → 409).
- Notification: idempotent `Submit`, route materialisation, push +
  email fan-out, `OnUserDeleted` cascade. Coverage of every catalog
  kind in `buildClientPushEvent` lives in
  `internal/notification/events_test.go`.
- Lobby: state-machine transitions, RND canonicalisation, sweeper.
- Runtime: per-game mutex serialisation, worker pool, scheduler,
  reconciler, force-next-turn skip flag.
- Admin: bcrypt cost 12, idempotent bootstrap, write-through cache,
  409 Conflict on duplicate username, last-used timestamp.
- Geo: counter increment on every authenticated request,
  declared-country write at registration, fail-open semantics.

### `galaxy/game`

The engine has its own service tests under `game/`:

- OpenAPI contract test (`game/openapi_contract_test.go`).
- Engine lifecycle (init, status, turn, banish, command, order,
  report) implemented by the engine package suites.

## Integration runbook

### Entry points

```bash
make -C integration preclean          # idempotent leftover cleanup
make -C integration integration       # preclean + serial test run
make -C integration integration-step  # preclean + one-test-at-a-time
```

`integration` runs every test in the module sequentially
(`-p=1 -parallel=1`) — recommended default on a slow / shared
Docker. `integration-step` runs them one at a time with a fresh
preclean before each test and stops on the first failure; useful to
isolate a flake or build up to a full pass without losing context to
subsequent tests.

### Why preclean matters

`preclean` keys off labels and removes:

- Containers labelled `org.testcontainers=true` (every container the
  testcontainers-go library brings up — backend, gateway, game,
  postgres, redis, mailpit, ryuk).
- Containers labelled `galaxy.backend=1` — engine instances spawned
  by backend's runtime adapter directly on the host Docker daemon
  (see `backend/internal/dockerclient/types.go`).
- Networks labelled `org.testcontainers=true`.
- Locally-built images labelled `galaxy.test.kind=integration-image`
  — the `galaxy/{backend,gateway,game}:integration` builds produced
  by `integration/testenv/images.go`. Pulled service images
  (`postgres:16-alpine`, `redis:7-alpine`, `axllent/mailpit`,
  `testcontainers/ryuk`) are **not** touched, so the cache stays
  warm.

### Ryuk reaper

The integration runners disable the testcontainers Ryuk reaper:

```makefile
export TESTCONTAINERS_RYUK_DISABLED = true
```

This is environment-driven, not principled — Ryuk does not start
cleanly on the local colima setup we use, and `preclean` covers the
same job by labels. Re-enable Ryuk by exporting
`TESTCONTAINERS_RYUK_DISABLED=false` (or unset) before invoking the
make target if you have an environment where Ryuk works.

### Cold runs

The first run after a clean checkout (or after `preclean`) rebuilds
three images: `galaxy/backend:integration`,
`galaxy/gateway:integration`, `galaxy/game:integration`. Cold cost
is ~30 s per image. Subsequent runs reuse the build cache; `preclean`
removes the tagged images themselves but BuildKit cache mounts
survive, so re-builds are fast.

## Integration test coverage

Mandatory inter-service coverage in `integration/`:

- **Gateway ↔ Backend (public auth)**:
  `auth_flow_test.go` — register + confirm with mailpit-captured
  code; declared_country populated; idempotent re-confirm.
- **Gateway ↔ Backend (authenticated user surface)**:
  `user_account_test.go`, `user_profile_update_test.go`,
  `user_settings_update_test.go` — signed envelope, FlatBuffers
  payload, response signature verification, BCP 47 / IANA validation.
- **Gateway ↔ Backend (anti-replay, signature, freshness)**:
  `gateway_edge_test.go` — body-too-large, bad signature,
  payload_hash mismatch, stale timestamp, unknown session,
  unsupported `protocol_version`.
- **Gateway ↔ Backend (push)**:
  `notification_flow_test.go`, `session_revoke_test.go` — push
  delivery to a SubscribeEvents stream and immediate stream close
  on revoke.
- **Gateway ↔ Backend (anti-replay)**:
  `anti_replay_test.go` — duplicate `request_id` rejected.
- **Backend ↔ Postgres** is exercised by every backend e2e test
  through testcontainers; integration tests do not duplicate it.
- **Backend ↔ SMTP**:
  `mail_flow_test.go` — login-code email captured by mailpit; admin
  list reaches `sent`; resend on `sent` returns 409.
- **Backend ↔ Game engine**:
  `runtime_lifecycle_test.go`, `engine_command_proxy_test.go` —
  start container, healthz green, command, force-next-turn, finish,
  race name promotion.
- **Admin surface (REST)**:
  `admin_flow_test.go`, `admin_global_games_view_test.go`,
  `admin_engine_versions_test.go`, `admin_user_sanction_test.go` —
  bootstrap + CRUD; visibility split between user and admin queries;
  engine-version registry CRUD; permanent block cascade.
- **Lobby flow without engine**:
  `lobby_flow_test.go` — owner-creates-private-game →
  open-enrollment → invite → redeem → memberships listing.
- **Soft delete cascade**:
  `soft_delete_test.go` — `POST /api/v1/user/account/delete`
  cascades through auth/lobby/notification/geo, gateway rejects
  subsequent calls.
- **Geo counters**:
  `geo_counter_increments_test.go` — multiple authenticated
  requests with different `X-Forwarded-For` values increment the
  user's per-country counter rows.

Full-system flows beyond the inter-service set are intentionally
limited; pick scenarios that exercise the longest vertical slice
the platform supports today.

## Principles

### Service tests

- **Postgres testcontainers must pin no-op observability providers.**
  Tests that call `pgshared.OpenPrimary(ctx, cfg)` from
  `galaxy/postgres` pass `backendpg.NoObservabilityOptions()...` so
  `otelsql` cannot fall through to the global tracer/meter providers.
  Without this, an unset OTEL endpoint in the developer environment
  can stall the test on a background exporter handshake.

  See `backend/internal/postgres/testopts.go` for the helper and
  `backend/internal/{auth,user,admin,lobby,mail,notification,runtime,geo,postgres}/`
  test files for the established call sites.

- **A bootstrap failure is fatal, not a skip.** A test that needs a
  testcontainer must fail loudly when the container fails to come
  up. `t.Skipf` is reserved for `testenv.RequireDocker` (no daemon
  at all); anything past that — `tcpostgres.Run`, `db.Ping`, schema
  migration — uses `t.Fatalf`.

### Integration tests

- **Bootstrap is per-test.** Each test calls `testenv.Bootstrap(t)`
  to spin up a dedicated Postgres, Redis, mailpit, backend, and
  gateway. Cross-test contamination is impossible.

- **Tests do not call `t.Parallel`.** Docker resource pressure makes
  parallel bootstraps flaky on commodity hardware.

- **Anti-abuse limits are loosened by `testenv/gateway.go`.** The
  bulk-scenario default lifts every gateway rate-limit class
  (`public_auth`, identity-bucket per-email, IP/session/user/
  message-class) to 10 000 req/window with a 1 000 burst. Negative-
  path edge tests in `gateway_edge_test.go` tighten specific limits
  per test to observe the protection firing.

- **Image labels are intentional.** `integration/testenv/images.go`
  stamps every locally-built image with
  `galaxy.test.kind=integration-image`; `preclean` keys off this
  label. Do not strip it from new image builds added to the test
  harness.

## Test file ownership matrix

| Suite                                      | Where             | Boots                                                                | Runs how                                  |
|--------------------------------------------|-------------------|----------------------------------------------------------------------|-------------------------------------------|
| `backend/internal/<pkg>/...` unit          | per package       | one Postgres testcontainer per test                                  | `go test ./internal/<pkg>/`               |
| `backend/push`                             | `backend/push/`   | nothing                                                              | `go test ./push/`                         |
| `gateway/internal/<pkg>/...` unit          | per package       | mostly nothing; few use redis tc                                     | `go test ./internal/<pkg>/`               |
| `pkg/transcoder`, `pkg/postgres` unit      | per package       | nothing / one tc per test                                            | `go test ./...` from the package          |
| `integration/`                             | `integration/`    | postgres + redis + mailpit + backend + gateway (+ optional game)     | `make -C integration integration`         |

## Adding a new test

1. Decide the layer: service, inter-service, or system. A backend
   change usually lands as service tests plus an integration test
   for any new cross-process behaviour.
2. Reuse `testenv` fixtures rather than rolling your own container
   orchestration.
3. Follow the bootstrap-per-test pattern; do not share a global
   stack across tests.
4. Make the test deterministic: explicit timeouts (no
   `time.Sleep`), `t.Logf` instead of `fmt.Println`, no
   `t.Parallel()` in `integration/`.
5. Service test that hits Postgres: copy the `startPostgres(t)`
   helper from one of the existing packages (e.g.
   `backend/internal/auth/auth_e2e_test.go`) and pass
   `backendpg.NoObservabilityOptions()...` to `pgshared.OpenPrimary`.
6. Integration test: add the file under `integration/`, call
   `testenv.Bootstrap(t)`, and use the typed clients exposed by
   `testenv` rather than reaching for raw HTTP. New scenarios that
   need bespoke gateway env should pass `Extra` through
   `BootstrapOptions` so the loosened defaults stay shared.
7. Any test that brings up its own Docker container (rare — most go
   through `testenv`) must label the container so `preclean` can
   find it on the next run.

## Day-to-day execution

- Run `go test ./<service>/...` for the service you are touching;
  this is fast (Postgres testcontainers add ~3–5 s per package that
  uses them).
- Run `make -C integration integration` before opening a PR that
  touches a cross-process seam. Cold runs build three Docker images
  (`galaxy/backend:integration`, `galaxy/gateway:integration`,
  `galaxy/game:integration`) — budget ~3 min for the cold path,
  ~75 s for the warm path.
- Use `make -C integration integration-step` when a flake or a real
  regression needs a per-test isolation pass.
- CI runs every layer on every push. Integration tests rely on a
  reachable Docker daemon; missing daemon yields a clear skip from
  `testenv.RequireDocker`, anything past that is a hard failure.

## Out-of-scope (legacy architecture)

The previous nine-service architecture defined components that no
longer exist as distinct services. Their behaviour either lives
inside `backend` (and is therefore covered by backend service or
integration tests) or has been removed:

- *Auth/Session Service*, *User Service*, *Notification Service*,
  *Mail Service*, *Game Lobby Service*, *Runtime Manager*,
  *Game Master*, *Admin Service* — consolidated into
  `backend/internal/*`. Inter-service seams between these former
  services are now in-process function calls; they are exercised by
  backend service tests, not by integration tests.
- *Geo Profile Service* (suspicious-multi-country detection,
  review-recommended state, session blocking through geo) — not
  implemented. The geo concern is intentionally minimal (see
  `ARCHITECTURE.md §10`) and the test plan does not assert on
  features we do not ship.
- *Billing Service* — not implemented; no tests required until it
  appears.