feat: runtime manager
This commit is contained in:
Ilia Denisov
+196
-16
@@ -658,12 +658,15 @@ It owns:
|
||||
* starting game engine containers;
|
||||
* stopping containers;
|
||||
* restarting containers where allowed;
|
||||
* patching/replacing containers where allowed;
|
||||
* patching/replacing containers (semver patch only) where allowed;
|
||||
* technical runtime inspection/status;
|
||||
* monitoring containers and publishing technical health events.
|
||||
* monitoring containers via Docker events, periodic inspect, and active HTTP probe;
|
||||
* publishing technical runtime events (`runtime:job_results`, `runtime:health_events`);
|
||||
* publishing admin-only notification intents for first-touch start failures.
|
||||
|
||||
It does **not** own platform metadata of games.
|
||||
It does **not** own runtime business state of games.
|
||||
It does **not** resolve engine versions; the producer (`Game Lobby` in v1, `Game Master` later) supplies `image_ref`.
|
||||
It executes runtime jobs for `Game Lobby` and `Game Master`.
|
||||
|
||||
### Container model
|
||||
@@ -673,6 +676,62 @@ It executes runtime jobs for `Game Lobby` and `Game Master`.
|
||||
|
||||
This is a hard invariant.
|
||||
|
||||
Each container is created with hostname `galaxy-game-{game_id}` and attached to the
|
||||
single user-defined Docker bridge network configured by `RTMANAGER_DOCKER_NETWORK`.
|
||||
The network is provisioned outside `Runtime Manager` (compose, Terraform, or operator
|
||||
runbook); a missing network is a fail-fast condition at startup. The published
|
||||
`engine_endpoint` is the stable URL `http://galaxy-game-{game_id}:8080`; restart and
|
||||
patch keep the same DNS name even though `current_container_id` changes.
|
||||
|
||||
### Image policy
|
||||
|
||||
`Runtime Manager` never resolves engine versions. The producer (`Game Lobby` in v1,
|
||||
`Game Master` once implemented) computes `image_ref` from its own template and
|
||||
hands it to `Runtime Manager` on the start envelope. `Runtime Manager` accepts the
|
||||
reference verbatim, applies the configured pull policy
|
||||
(`RTMANAGER_IMAGE_PULL_POLICY`), and reads container resource limits from labels
|
||||
on the resolved image.
|
||||
|
||||
The producer-supplied `image_ref` rule decouples `Runtime Manager` from any
|
||||
engine-version arbitration logic, lets the v1 launch ship without `Game Master`'s
|
||||
engine-version registry, and cleanly separates "which image to run" (Lobby/GM
|
||||
concern) from "how to run it" (RTM concern). Two alternatives were rejected:
|
||||
RTM holding its own image map (would need to consume upstream tariff or
|
||||
compatibility signals that belong in the producers) and RTM resolving the
|
||||
image at start time by querying GM (would create a circular dependency for
|
||||
v1 and add a synchronous hop on the hot path).
|
||||
|
||||
Patch is restart with a new `image_ref` and is allowed only as a semver patch
|
||||
within the same major/minor line; cross-major or cross-minor patch attempts fail
|
||||
with `semver_patch_only`. Producers that need to change the major/minor line must
|
||||
stop the game and start a new container.
|
||||
|
||||
### State ownership
|
||||
|
||||
Engine state lives on the host filesystem under the per-game directory
|
||||
`<RTMANAGER_GAME_STATE_ROOT>/{game_id}` and is bind-mounted into the container at
|
||||
`RTMANAGER_ENGINE_STATE_MOUNT_PATH`. The mount path is exposed to the engine through
|
||||
`GAME_STATE_PATH` and, for backward compatibility, also as `STORAGE_PATH`. Both
|
||||
names are accepted by `galaxy/game` in v1.
|
||||
|
||||
`Runtime Manager` never deletes the host state directory. Removing a container
|
||||
through the cleanup endpoint or the retention TTL leaves the directory intact.
|
||||
Backup, archival, and operator cleanup of state directories belong to operator
|
||||
tooling or a future Admin Service workflow.
|
||||
|
||||
### Reconcile policy
|
||||
|
||||
`Runtime Manager` reconciles its `runtime_records` with Docker reality at startup
|
||||
(blocking, before workers start) and on a periodic interval
|
||||
(`RTMANAGER_RECONCILE_INTERVAL`). Two rules apply unconditionally:
|
||||
|
||||
* unrecorded containers labelled `com.galaxy.owner=rtmanager` are **adopted** into
|
||||
`runtime_records` as `running`, never killed; operators may have launched one
|
||||
manually for diagnostics;
|
||||
* recorded `running` rows whose container is missing in Docker are marked
|
||||
`removed`, with a `container_disappeared` event emitted on
|
||||
`runtime:health_events`.
|
||||
|
||||
## 10. [Notification Service](notification/README.md)
|
||||
|
||||
`Notification Service` is the async delivery/orchestration layer for platform notifications.
|
||||
@@ -770,6 +829,18 @@ The platform uses one simple rule:
|
||||
* if the user-facing request must complete with a deterministic result in the same flow, the critical internal chain is synchronous;
|
||||
* if the interaction is propagation, notification, cache invalidation, runtime job completion, telemetry, or denormalized read-model update, it is asynchronous.
|
||||
|
||||
The `Lobby ↔ Runtime Manager` transport is the canonical asynchronous case:
|
||||
Lobby drives RTM exclusively through Redis Streams (`runtime:start_jobs`,
|
||||
`runtime:stop_jobs`, `runtime:job_results`); there is no synchronous
|
||||
Lobby→RTM REST call in v1, and no plan to add one. Synchronous coupling
|
||||
would force Lobby to block on Docker pull/start latency, which is
|
||||
unbounded in the worst case. `Game Master` and `Admin Service`, by contrast,
|
||||
drive RTM synchronously over REST because they operate on already-running
|
||||
containers and need deterministic per-request outcomes (for example,
|
||||
"restart this game's container now"); routing those operations through
|
||||
streams would force operators to correlate async results back to admin
|
||||
requests for no operational benefit.
|
||||
|
||||
### Fixed synchronous interactions
|
||||
|
||||
* `Gateway -> Auth / Session Service`
|
||||
@@ -783,13 +854,17 @@ The platform uses one simple rule:
|
||||
* `Geo Profile Service -> User Service`
|
||||
* `Game Lobby -> User Service`
|
||||
* `Game Lobby -> Game Master` for critical registration/update calls
|
||||
* `Game Master -> Runtime Manager` for inspect, restart, patch, stop, and cleanup REST calls
|
||||
* `Admin Service -> Runtime Manager` for operational inspect, restart, patch, stop, and cleanup REST calls
|
||||
|
||||
### Fixed asynchronous interactions
|
||||
|
||||
* session lifecycle projection toward gateway cache;
|
||||
* revoke propagation;
|
||||
* `Lobby -> Runtime Manager` runtime jobs;
|
||||
* `Game Master -> Runtime Manager` runtime jobs;
|
||||
* `Lobby -> Runtime Manager` runtime jobs through `runtime:start_jobs` (`{game_id, image_ref, requested_at_ms}`) and `runtime:stop_jobs` (`{game_id, reason, requested_at_ms}`);
|
||||
* `Runtime Manager -> Lobby` job outcomes through `runtime:job_results`;
|
||||
* `Runtime Manager -> Notification Service` admin-only failure intents (image pull, container start, start config) through `notification:intents`;
|
||||
* `Runtime Manager` outbound technical health stream `runtime:health_events` consumed by `Game Master`; `Game Lobby` and `Admin Service` are reserved as future consumers;
|
||||
* all event-bus propagation;
|
||||
* `Game Master -> Game Lobby` runtime snapshot updates (including
|
||||
`player_turn_stats` for capability aggregation) and game-finish events
|
||||
@@ -831,6 +906,8 @@ PostgreSQL is the source of truth for table-shaped business state:
|
||||
malformed-intent audit;
|
||||
* lobby games, applications, invites, memberships, and the race-name
|
||||
registry (registered/reservation/pending tiers);
|
||||
* runtime manager runtime records (`game_id -> current_container_id`),
|
||||
per-operation audit log, and latest health snapshot per game;
|
||||
* idempotency records, expressed as `UNIQUE` constraints on the durable
|
||||
table — not as a separate kv;
|
||||
* retry scheduling state, expressed as a `next_attempt_at` column on the
|
||||
@@ -839,11 +916,13 @@ PostgreSQL is the source of truth for table-shaped business state:
|
||||
Redis is the source of truth for ephemeral and runtime-coordination state:
|
||||
|
||||
* the platform event bus implemented as Redis Streams (`user:domain_events`,
|
||||
`user:lifecycle_events`, `gm:lobby_events`, `runtime:job_results`,
|
||||
`user:lifecycle_events`, `gm:lobby_events`, `runtime:start_jobs`,
|
||||
`runtime:stop_jobs`, `runtime:job_results`, `runtime:health_events`,
|
||||
`notification:intents`, `gateway:client-events`, `mail:delivery_commands`);
|
||||
* stream consumer offsets;
|
||||
* gateway session cache, replay reservations, rate-limit counters, and
|
||||
short-lived runtime locks/leases (e.g. notification `route_leases`);
|
||||
short-lived runtime locks/leases (e.g. notification `route_leases`,
|
||||
runtime manager per-game operation leases `rtmanager:game_lease:{game_id}`);
|
||||
* `Auth / Session Service` challenges and active session tokens, which are
|
||||
TTL-bounded and where loss is recoverable by re-authentication;
|
||||
* lobby per-game runtime aggregates that are deleted at game finish
|
||||
@@ -852,9 +931,9 @@ Redis is the source of truth for ephemeral and runtime-coordination state:
|
||||
### Database topology
|
||||
|
||||
* Single PostgreSQL database `galaxy`.
|
||||
* Schema per service: `user`, `mail`, `notification`, `lobby`. Reserved for
|
||||
future use: `geoprofile`. Not allocated unless needed: `gateway`,
|
||||
`authsession`.
|
||||
* Schema per service: `user`, `mail`, `notification`, `lobby`, `rtmanager`.
|
||||
Reserved for future use: `geoprofile`. Not allocated unless needed:
|
||||
`gateway`, `authsession`.
|
||||
* Each service connects with its own PostgreSQL role whose grants are
|
||||
restricted to its own schema (defense-in-depth).
|
||||
* Authentication is username + password only. `sslmode=disable`. No client
|
||||
@@ -933,15 +1012,15 @@ crossing the SQL boundary carry `time.UTC` as their location.
|
||||
### Configuration
|
||||
|
||||
For each service `<S>` ∈ { `USERSERVICE`, `MAIL`, `NOTIFICATION`,
|
||||
`LOBBY`, `GATEWAY`, `AUTHSESSION` }, the Redis connection accepts:
|
||||
`LOBBY`, `RTMANAGER`, `GATEWAY`, `AUTHSESSION` }, the Redis connection accepts:
|
||||
|
||||
* `<S>_REDIS_MASTER_ADDR` (required)
|
||||
* `<S>_REDIS_REPLICA_ADDRS` (optional, comma-separated)
|
||||
* `<S>_REDIS_PASSWORD` (required)
|
||||
* `<S>_REDIS_DB`, `<S>_REDIS_OPERATION_TIMEOUT`
|
||||
|
||||
For PG-backed services (`USERSERVICE`, `MAIL`, `NOTIFICATION`, `LOBBY`)
|
||||
the Postgres connection accepts:
|
||||
For PG-backed services (`USERSERVICE`, `MAIL`, `NOTIFICATION`, `LOBBY`,
|
||||
`RTMANAGER`) the Postgres connection accepts:
|
||||
|
||||
* `<S>_POSTGRES_PRIMARY_DSN` (required;
|
||||
`postgres://<role>:<pwd>@<host>:5432/galaxy?search_path=<schema>&sslmode=disable`)
|
||||
@@ -951,9 +1030,105 @@ the Postgres connection accepts:
|
||||
|
||||
Stream- and key-shape env vars (`*_REDIS_DOMAIN_EVENTS_STREAM`,
|
||||
`*_REDIS_LIFECYCLE_EVENTS_STREAM`, `*_REDIS_KEYSPACE_PREFIX`,
|
||||
`MAIL_REDIS_COMMAND_STREAM`, `NOTIFICATION_INTENTS_STREAM`, etc.) keep
|
||||
their current names and semantics — they describe stream/key shapes, not
|
||||
connection topology.
|
||||
`MAIL_REDIS_COMMAND_STREAM`, `NOTIFICATION_INTENTS_STREAM`,
|
||||
`RTMANAGER_REDIS_START_JOBS_STREAM`, `RTMANAGER_REDIS_STOP_JOBS_STREAM`,
|
||||
`RTMANAGER_REDIS_JOB_RESULTS_STREAM`, `RTMANAGER_REDIS_HEALTH_EVENTS_STREAM`,
|
||||
etc.) keep their current names and semantics — they describe stream/key
|
||||
shapes, not connection topology.
|
||||
|
||||
## Test and Contract Conventions
|
||||
|
||||
The repository follows a small set of cross-service rules for contract
|
||||
specifications and test doubles. Each rule is captured below with the
|
||||
rejected alternatives so future services do not re-litigate them.
|
||||
|
||||
### AsyncAPI version: 3.1.0
|
||||
|
||||
Every AsyncAPI spec in the repository declares `asyncapi: 3.1.0`
|
||||
(`notification/api/intents-asyncapi.yaml`,
|
||||
`rtmanager/api/runtime-jobs-asyncapi.yaml`,
|
||||
`rtmanager/api/runtime-health-asyncapi.yaml`). Operators read the same
|
||||
shape across services — channel with `address`, separate `operations`
|
||||
block, `action: send | receive` vocabulary.
|
||||
|
||||
Alternatives rejected:
|
||||
|
||||
- AsyncAPI 2.6.0 — would carry the same information under different
|
||||
field names (`publish` / `subscribe` blocks living inside the channel)
|
||||
and the shared YAML walker assertions would not transfer cleanly;
|
||||
- adding a typed AsyncAPI parser library — no Galaxy service uses one
|
||||
today; introducing a new dependency for the existing specs would
|
||||
break the established pattern that all AsyncAPI freeze tests are pure
|
||||
YAML walkers using `gopkg.in/yaml.v3`.
|
||||
|
||||
The `oneOf`-based polymorphism on the `details` field in
|
||||
`runtime-health-asyncapi.yaml` is plain JSON Schema and works
|
||||
identically in 3.1.0; no AsyncAPI-version-specific feature is used. If
|
||||
`notification/api/intents-asyncapi.yaml` ever moves to a newer major,
|
||||
every downstream service moves with it as a cross-service contract bump.
|
||||
|
||||
### Contract freeze tests
|
||||
|
||||
OpenAPI freeze tests use `github.com/getkin/kin-openapi/openapi3`. The
|
||||
library is already a workspace-wide dependency
|
||||
(`lobby/contract_openapi_test.go`, `game/openapi_contract_test.go`,
|
||||
`rtmanager/contract_openapi_test.go`). It validates OpenAPI 3.0
|
||||
syntactic correctness, exposes a typed AST, and lets assertions reach
|
||||
operation IDs, schema references, required fields, and enum membership
|
||||
without a hand-rolled parser.
|
||||
|
||||
AsyncAPI freeze tests use `gopkg.in/yaml.v3` plus a small set of
|
||||
helpers (`getMapValue`, `getStringValue`, `getStringSlice`,
|
||||
`getSliceValue`, `getBoolValue`). AsyncAPI 3.1.0 is itself a JSON
|
||||
Schema document; the freeze tests only need to assert on field paths,
|
||||
enum membership, required fields, and `$ref` targets — none of which
|
||||
require type-aware parsing.
|
||||
|
||||
Both freeze tests live at the module root (`package <service>` next to
|
||||
`go.mod`) for every service. A subpackage like `<service>/contracts/`
|
||||
would have to import the service's domain types to share constants,
|
||||
which would create the exact import cycle the freeze tests are meant
|
||||
to prevent.
|
||||
|
||||
### Test doubles: `mockgen` for narrow recorder ports, `*inmem` for behavioural fakes
|
||||
|
||||
Test doubles in the repository follow a three-track convention:
|
||||
|
||||
- **Narrow recorder ports** (interfaces whose implementation has no
|
||||
domain semantics — record calls, return injectable errors, expose
|
||||
accessor methods) use `go.uber.org/mock` mocks. Examples:
|
||||
`lobby/internal/ports/{RuntimeManager, IntentPublisher, GMClient,
|
||||
UserService}`, `rtmanager/internal/ports/DockerClient`,
|
||||
`rtmanager/internal/api/internalhttp/handlers/{Start,Stop,Restart,
|
||||
Patch,Cleanup}Service`. `//go:generate` directives live next to the
|
||||
interface declaration; generated mocks are committed under
|
||||
`<module>/internal/adapters/mocks/` (or `handlers/mocks/`); the
|
||||
`make -C <module> mocks` target regenerates them.
|
||||
- **Behavioural in-memory adapters** (re-implement the production
|
||||
contract — CAS, domain transitions, monotonic invariants, two-tier
|
||||
invariants like the Race Name Directory) live under
|
||||
`<module>/internal/adapters/<thing>inmem/` and stay hand-rolled.
|
||||
Replacing them with `mockgen` would force every consumer site to
|
||||
script `EXPECT()` chains for behaviour the fake currently handles
|
||||
automatically, and would lose the cross-implementation parity guarantee.
|
||||
- **Dead test doubles** with no consumers are deleted on sight.
|
||||
|
||||
Per-test recorder helpers (small structs holding captured slices and
|
||||
per-test error injection) live **inside the test files that use them**
|
||||
rather than in a shared `mockrec` / `testfixtures` package. A shared
|
||||
package would re-create the retired `*stub` convention in a different
|
||||
namespace; per-test recorders are easy to specialise without polluting
|
||||
a shared surface.
|
||||
|
||||
`racenameinmem` is a special case: it is also one of two selectable
|
||||
Race Name Directory backends chosen via
|
||||
`LOBBY_RACE_NAME_DIRECTORY_BACKEND=stub` (the config token name is
|
||||
preserved while the package name follows the `*inmem` convention; both
|
||||
backends pass the shared conformance suite at
|
||||
`lobby/internal/ports/racenamedirtest/`).
|
||||
|
||||
The maintained `go.uber.org/mock` fork is preferred over the archived
|
||||
`github.com/golang/mock`.
|
||||
|
||||
## Main End-to-End Flows
|
||||
|
||||
@@ -1283,7 +1458,12 @@ Recommended order for implementation is:
|
||||
Platform game records, membership, invites, applications, approvals, schedules, user-facing lists, pre-start lifecycle.
|
||||
|
||||
7. **Runtime Manager**
|
||||
Dedicated Docker-control service for container start/stop/patch/status and technical runtime monitoring.
|
||||
Dedicated Docker-control service for container lifecycle (start, stop,
|
||||
restart, semver-patch, cleanup) and inspect/health monitoring through
|
||||
Docker events, periodic inspect, and active HTTP probes. Driven
|
||||
asynchronously from `Game Lobby` via `runtime:start_jobs` /
|
||||
`runtime:stop_jobs` and synchronously from `Game Master` and
|
||||
`Admin Service` via the trusted internal REST surface.
|
||||
|
||||
8. **Game Master**
|
||||
Running-game orchestration, engine version registry, runtime state, turn scheduler, engine API mediation, operational controls.
|
||||
|
||||
Reference in New Issue
Block a user