Ilia Denisov
334 lines
16 KiB
Markdown
334 lines
16 KiB
Markdown
# Testing
|
||
|
||
Test strategy and runbook for the [Galaxy Game](ARCHITECTURE.md)
|
||
platform. The platform ships three executables — `gateway`,
|
||
`backend`, `game` (the engine container) — plus the shared `pkg/*`
|
||
libraries. This document defines the layering of tests, the
|
||
mandatory minimum coverage per executable, the integration runbook,
|
||
and the principles every test must follow.
|
||
|
||
## Layers
|
||
|
||
1. **Service tests** verify a single executable in isolation. They
|
||
live next to the implementation as `*_test.go` files and use only
|
||
in-process or testcontainers-managed dependencies. The package
|
||
either runs entirely in process or boots a single Postgres
|
||
testcontainer per test.
|
||
2. **Inter-service integration tests** verify one cross-process seam
|
||
between two real executables (most often `gateway ↔ backend`,
|
||
sometimes `backend ↔ game`). They live in
|
||
[`galaxy/integration/`](../integration/) and drive the platform
|
||
from outside the trust boundary.
|
||
3. **Full system tests** are a small, focused subset of the
|
||
integration suite that walks an entire user-facing flow from the
|
||
client edge through every component the flow touches. They live
|
||
in the same `integration/` module and reuse the same fixtures.
|
||
|
||
Service tests are the cheapest and the broadest; integration tests
|
||
are slower and broader; full-system tests are the slowest and the
|
||
narrowest. The pyramid stays in this order — never replace a service
|
||
test with a system test.
|
||
|
||
## Global rules
|
||
|
||
- Every executable owns the service tests for its packages. Adding a
|
||
new package without `_test.go` files is a review block.
|
||
- Every cross-process seam must have at least one passing
|
||
inter-service test before the seam is wired in production.
|
||
- Async flows (mail outbox, notification routes, runtime workers,
|
||
push gRPC) get tests for both the success path and the retry /
|
||
dead-letter path, and a duplicate-event safety check.
|
||
- Sync flows get happy path, validation failure, timeout
|
||
propagation, and dependency unavailable.
|
||
- Every external or trusted-internal API must have contract tests
|
||
alongside behaviour tests. `backend/internal/server/contract_test.go`
|
||
is the reference; gateway runs the same shape against
|
||
`gateway/openapi.yaml`.
|
||
- The integration suite must keep running on a developer machine
|
||
with Docker available. The only acceptable `t.Skip` is
|
||
`testenv.RequireDocker` (no daemon at all). Any failure deeper
|
||
than that — `tcpostgres.Run`, network create, image build, schema
|
||
migration — fails the test loudly with `t.Fatal`. The historical
|
||
bug we fixed (silent skips on reaper failures masking 27
|
||
integration tests as "ok") came from treating an environment
|
||
break as a skip.
|
||
|
||
## Service-specific coverage
|
||
|
||
### `galaxy/gateway`
|
||
|
||
Service tests live under `gateway/internal/`:
|
||
|
||
- Public REST routing, error projection, and OpenAPI contract
|
||
validation.
|
||
- Authenticated gRPC envelope verification (`grpcapi.Server`):
|
||
signature, payload hash, freshness window, anti-replay reservation,
|
||
unknown / revoked sessions.
|
||
- Session cache (`session.BackendCache`) — the only implementation
|
||
in the codebase, a thin wrapper around the `backendclient.RESTClient`
|
||
per-request lookup.
|
||
- Response signing for unary responses and stream events
|
||
(`authn.ResponseSigner`).
|
||
- Push hub (`push.Hub`) and push fan-out (`push_fanout.go`).
|
||
- Replay store (`replay.RedisStore`) reservation semantics.
|
||
- Anti-abuse rate limits per IP / session / user / message class.
|
||
|
||
### `galaxy/backend`
|
||
|
||
Service tests live under `backend/internal/`:
|
||
|
||
- Startup wiring: `app.App` lifecycle, telemetry runtime, Postgres
|
||
pool, embedded migrations.
|
||
- OpenAPI contract test (`internal/server/contract_test.go`):
|
||
validates every documented operation against the live gin engine.
|
||
- Domain unit + e2e tests per package (`auth`, `user`, `admin`,
|
||
`lobby`, `runtime`, `mail`, `notification`, `geo`, `push`).
|
||
E2E tests (`*_e2e_test.go`) spin up a Postgres testcontainer.
|
||
- Mail outbox: pickup with `SELECT FOR UPDATE SKIP LOCKED`, retry
|
||
with backoff plus jitter, dead-letter past `MAX_ATTEMPTS`,
|
||
resend semantics (`pending|retrying|dead_lettered` → re-armed,
|
||
`sent` → 409).
|
||
- Notification: idempotent `Submit`, route materialisation, push +
|
||
email fan-out, `OnUserDeleted` cascade. Coverage of every catalog
|
||
kind in `buildClientPushEvent` lives in
|
||
`internal/notification/events_test.go`.
|
||
- Lobby: state-machine transitions, RND canonicalisation, sweeper.
|
||
- Runtime: per-game mutex serialisation, worker pool, scheduler,
|
||
reconciler, force-next-turn skip flag.
|
||
- Admin: bcrypt cost 12, idempotent bootstrap, write-through cache,
|
||
409 Conflict on duplicate username, last-used timestamp.
|
||
- Geo: counter increment on every authenticated request,
|
||
declared-country write at registration, fail-open semantics.
|
||
|
||
### `galaxy/game`
|
||
|
||
The engine has its own service tests under `game/`:
|
||
|
||
- OpenAPI contract test (`game/openapi_contract_test.go`).
|
||
- Engine lifecycle (init, status, turn, banish, command, order,
|
||
report) implemented by the engine package suites.
|
||
|
||
## Integration runbook
|
||
|
||
### Entry points
|
||
|
||
```bash
|
||
make -C integration preclean # idempotent leftover cleanup
|
||
make -C integration integration # preclean + serial test run
|
||
make -C integration integration-step # preclean + one-test-at-a-time
|
||
```
|
||
|
||
`integration` runs every test in the module sequentially
|
||
(`-p=1 -parallel=1`) — recommended default on a slow / shared
|
||
Docker. `integration-step` runs them one at a time with a fresh
|
||
preclean before each test and stops on the first failure; useful to
|
||
isolate a flake or build up to a full pass without losing context to
|
||
subsequent tests.
|
||
|
||
### Why preclean matters
|
||
|
||
`preclean` keys off labels and removes:
|
||
|
||
- Containers labelled `org.testcontainers=true` (every container the
|
||
testcontainers-go library brings up — backend, gateway, game,
|
||
postgres, redis, mailpit, ryuk).
|
||
- Containers labelled `galaxy.backend=1` — engine instances spawned
|
||
by backend's runtime adapter directly on the host Docker daemon
|
||
(see `backend/internal/dockerclient/types.go`).
|
||
- Networks labelled `org.testcontainers=true`.
|
||
- Locally-built images labelled `galaxy.test.kind=integration-image`
|
||
— the `galaxy/{backend,gateway,game}:integration` builds produced
|
||
by `integration/testenv/images.go`. Pulled service images
|
||
(`postgres:16-alpine`, `redis:7-alpine`, `axllent/mailpit`,
|
||
`testcontainers/ryuk`) are **not** touched, so the cache stays
|
||
warm.
|
||
|
||
### Ryuk reaper
|
||
|
||
The integration runners disable the testcontainers Ryuk reaper:
|
||
|
||
```makefile
|
||
export TESTCONTAINERS_RYUK_DISABLED = true
|
||
```
|
||
|
||
This is environment-driven, not principled — Ryuk does not start
|
||
cleanly on the local colima setup we use, and `preclean` covers the
|
||
same job by labels. Re-enable Ryuk by exporting
|
||
`TESTCONTAINERS_RYUK_DISABLED=false` (or unset) before invoking the
|
||
make target if you have an environment where Ryuk works.
|
||
|
||
### Cold runs
|
||
|
||
The first run after a clean checkout (or after `preclean`) rebuilds
|
||
three images: `galaxy/backend:integration`,
|
||
`galaxy/gateway:integration`, `galaxy/game:integration`. Cold cost
|
||
is ~30 s per image. Subsequent runs reuse the build cache; `preclean`
|
||
removes the tagged images themselves but BuildKit cache mounts
|
||
survive, so re-builds are fast.
|
||
|
||
## Integration test coverage
|
||
|
||
Mandatory inter-service coverage in `integration/`:
|
||
|
||
- **Gateway ↔ Backend (public auth)**:
|
||
`auth_flow_test.go` — register + confirm with mailpit-captured
|
||
code; declared_country populated; idempotent re-confirm.
|
||
- **Gateway ↔ Backend (authenticated user surface)**:
|
||
`user_account_test.go`, `user_profile_update_test.go`,
|
||
`user_settings_update_test.go` — signed envelope, FlatBuffers
|
||
payload, response signature verification, BCP 47 / IANA validation.
|
||
- **Gateway ↔ Backend (anti-replay, signature, freshness)**:
|
||
`gateway_edge_test.go` — body-too-large, bad signature,
|
||
payload_hash mismatch, stale timestamp, unknown session,
|
||
unsupported `protocol_version`.
|
||
- **Gateway ↔ Backend (push)**:
|
||
`notification_flow_test.go`, `session_revoke_test.go` — push
|
||
delivery to a SubscribeEvents stream and immediate stream close
|
||
on revoke.
|
||
- **Gateway ↔ Backend (anti-replay)**:
|
||
`anti_replay_test.go` — duplicate `request_id` rejected.
|
||
- **Backend ↔ Postgres** is exercised by every backend e2e test
|
||
through testcontainers; integration tests do not duplicate it.
|
||
- **Backend ↔ SMTP**:
|
||
`mail_flow_test.go` — login-code email captured by mailpit; admin
|
||
list reaches `sent`; resend on `sent` returns 409.
|
||
- **Backend ↔ Game engine**:
|
||
`runtime_lifecycle_test.go`, `engine_command_proxy_test.go` —
|
||
start container, healthz green, command, force-next-turn, finish,
|
||
race name promotion.
|
||
- **Admin surface (REST)**:
|
||
`admin_flow_test.go`, `admin_global_games_view_test.go`,
|
||
`admin_engine_versions_test.go`, `admin_user_sanction_test.go` —
|
||
bootstrap + CRUD; visibility split between user and admin queries;
|
||
engine-version registry CRUD; permanent block cascade.
|
||
- **Lobby flow without engine**:
|
||
`lobby_flow_test.go` — owner-creates-private-game →
|
||
open-enrollment → invite → redeem → memberships listing.
|
||
- **Soft delete cascade**:
|
||
`soft_delete_test.go` — `POST /api/v1/user/account/delete`
|
||
cascades through auth/lobby/notification/geo, gateway rejects
|
||
subsequent calls.
|
||
- **Geo counters**:
|
||
`geo_counter_increments_test.go` — multiple authenticated
|
||
requests with different `X-Forwarded-For` values increment the
|
||
user's per-country counter rows.
|
||
|
||
Full-system flows beyond the inter-service set are intentionally
|
||
limited; pick scenarios that exercise the longest vertical slice
|
||
the platform supports today.
|
||
|
||
## Principles
|
||
|
||
### Service tests
|
||
|
||
- **Postgres testcontainers must pin no-op observability providers.**
|
||
Tests that call `pgshared.OpenPrimary(ctx, cfg)` from
|
||
`galaxy/postgres` pass `backendpg.NoObservabilityOptions()...` so
|
||
`otelsql` cannot fall through to the global tracer/meter providers.
|
||
Without this, an unset OTEL endpoint in the developer environment
|
||
can stall the test on a background exporter handshake.
|
||
|
||
See `backend/internal/postgres/testopts.go` for the helper and
|
||
`backend/internal/{auth,user,admin,lobby,mail,notification,runtime,geo,postgres}/`
|
||
test files for the established call sites.
|
||
|
||
- **A bootstrap failure is fatal, not a skip.** A test that needs a
|
||
testcontainer must fail loudly when the container fails to come
|
||
up. `t.Skipf` is reserved for `testenv.RequireDocker` (no daemon
|
||
at all); anything past that — `tcpostgres.Run`, `db.Ping`, schema
|
||
migration — uses `t.Fatalf`.
|
||
|
||
### Integration tests
|
||
|
||
- **Bootstrap is per-test.** Each test calls `testenv.Bootstrap(t)`
|
||
to spin up a dedicated Postgres, Redis, mailpit, backend, and
|
||
gateway. Cross-test contamination is impossible.
|
||
|
||
- **Tests do not call `t.Parallel`.** Docker resource pressure makes
|
||
parallel bootstraps flaky on commodity hardware.
|
||
|
||
- **Anti-abuse limits are loosened by `testenv/gateway.go`.** The
|
||
bulk-scenario default lifts every gateway rate-limit class
|
||
(`public_auth`, identity-bucket per-email, IP/session/user/
|
||
message-class) to 10 000 req/window with a 1 000 burst. Negative-
|
||
path edge tests in `gateway_edge_test.go` tighten specific limits
|
||
per test to observe the protection firing.
|
||
|
||
- **Image labels are intentional.** `integration/testenv/images.go`
|
||
stamps every locally-built image with
|
||
`galaxy.test.kind=integration-image`; `preclean` keys off this
|
||
label. Do not strip it from new image builds added to the test
|
||
harness.
|
||
|
||
## Test file ownership matrix
|
||
|
||
| Suite | Where | Boots | Runs how |
|
||
|--------------------------------------------|-------------------|----------------------------------------------------------------------|-------------------------------------------|
|
||
| `backend/internal/<pkg>/...` unit | per package | one Postgres testcontainer per test | `go test ./internal/<pkg>/` |
|
||
| `backend/push` | `backend/push/` | nothing | `go test ./push/` |
|
||
| `gateway/internal/<pkg>/...` unit | per package | mostly nothing; few use redis tc | `go test ./internal/<pkg>/` |
|
||
| `pkg/transcoder`, `pkg/postgres` unit | per package | nothing / one tc per test | `go test ./...` from the package |
|
||
| `integration/` | `integration/` | postgres + redis + mailpit + backend + gateway (+ optional game) | `make -C integration integration` |
|
||
|
||
## Adding a new test
|
||
|
||
1. Decide the layer: service, inter-service, or system. A backend
|
||
change usually lands as service tests plus an integration test
|
||
for any new cross-process behaviour.
|
||
2. Reuse `testenv` fixtures rather than rolling your own container
|
||
orchestration.
|
||
3. Follow the bootstrap-per-test pattern; do not share a global
|
||
stack across tests.
|
||
4. Make the test deterministic: explicit timeouts (no
|
||
`time.Sleep`), `t.Logf` instead of `fmt.Println`, no
|
||
`t.Parallel()` in `integration/`.
|
||
5. Service test that hits Postgres: copy the `startPostgres(t)`
|
||
helper from one of the existing packages (e.g.
|
||
`backend/internal/auth/auth_e2e_test.go`) and pass
|
||
`backendpg.NoObservabilityOptions()...` to `pgshared.OpenPrimary`.
|
||
6. Integration test: add the file under `integration/`, call
|
||
`testenv.Bootstrap(t)`, and use the typed clients exposed by
|
||
`testenv` rather than reaching for raw HTTP. New scenarios that
|
||
need bespoke gateway env should pass `Extra` through
|
||
`BootstrapOptions` so the loosened defaults stay shared.
|
||
7. Any test that brings up its own Docker container (rare — most go
|
||
through `testenv`) must label the container so `preclean` can
|
||
find it on the next run.
|
||
|
||
## Day-to-day execution
|
||
|
||
- Run `go test ./<service>/...` for the service you are touching;
|
||
this is fast (Postgres testcontainers add ~3–5 s per package that
|
||
uses them).
|
||
- Run `make -C integration integration` before opening a PR that
|
||
touches a cross-process seam. Cold runs build three Docker images
|
||
(`galaxy/backend:integration`, `galaxy/gateway:integration`,
|
||
`galaxy/game:integration`) — budget ~3 min for the cold path,
|
||
~75 s for the warm path.
|
||
- Use `make -C integration integration-step` when a flake or a real
|
||
regression needs a per-test isolation pass.
|
||
- CI runs every layer on every push. Integration tests rely on a
|
||
reachable Docker daemon; missing daemon yields a clear skip from
|
||
`testenv.RequireDocker`, anything past that is a hard failure.
|
||
|
||
## Out-of-scope (legacy architecture)
|
||
|
||
The previous nine-service architecture defined components that no
|
||
longer exist as distinct services. Their behaviour either lives
|
||
inside `backend` (and is therefore covered by backend service or
|
||
integration tests) or has been removed:
|
||
|
||
- *Auth/Session Service*, *User Service*, *Notification Service*,
|
||
*Mail Service*, *Game Lobby Service*, *Runtime Manager*,
|
||
*Game Master*, *Admin Service* — consolidated into
|
||
`backend/internal/*`. Inter-service seams between these former
|
||
services are now in-process function calls; they are exercised by
|
||
backend service tests, not by integration tests.
|
||
- *Geo Profile Service* (suspicious-multi-country detection,
|
||
review-recommended state, session blocking through geo) — not
|
||
implemented. The geo concern is intentionally minimal (see
|
||
`ARCHITECTURE.md §10`) and the test plan does not assert on
|
||
features we do not ship.
|
||
- *Billing Service* — not implemented; no tests required until it
|
||
appears.
|