galaxy-game/backend/docs/runtime.md

Runtime and Components

The diagram below focuses on the deployed galaxy/backend process and its runtime dependencies. Every component is wired in backend/cmd/backend/main.go.

flowchart LR
    subgraph Inbound
        Gateway["Gateway<br/>HTTP + gRPC push subscriber"]
        Probes["Liveness / readiness<br/>probes"]
    end

    subgraph BackendProcess["Backend process"]
        HTTP["HTTP listener<br/>:8080<br/>/api/v1/{public,user,internal,admin}"]
        Push["gRPC push listener<br/>:8081<br/>Push.SubscribePush"]
        Metrics["Optional Prometheus<br/>metrics listener"]
        AuthSvc["auth.Service"]
        UserSvc["user.Service"]
        AdminSvc["admin.Service"]
        LobbySvc["lobby.Service"]
        RuntimeSvc["runtime.Service"]
        MailSvc["mail.Service"]
        NotifSvc["notification.Service"]
        GeoSvc["geo.Service"]
        PushSvc["push.Service<br/>(ring buffer + cursor)"]
        Caches["Write-through caches<br/>auth / user / admin /<br/>lobby / runtime"]
        MailWorker["mail worker"]
        NotifWorker["notification worker"]
        Sweeper["lobby sweeper"]
        RuntimeWorkers["runtime worker pool +<br/>scheduler + reconciler"]
        Telemetry["zap + OpenTelemetry"]
    end

    Postgres[(Postgres<br/>backend schema)]
    Docker[(Docker daemon)]
    SMTP[(SMTP relay)]
    GeoDB[(GeoLite2 mmdb)]
    Game[(galaxy-game-{id}<br/>engine containers)]

    Gateway --> HTTP
    Gateway --> Push
    Probes --> HTTP

    HTTP --> AuthSvc & UserSvc & AdminSvc & LobbySvc & RuntimeSvc & MailSvc & NotifSvc & GeoSvc
    Push --> PushSvc

    AuthSvc & UserSvc & AdminSvc & LobbySvc & RuntimeSvc & MailSvc & NotifSvc --> Caches
    AuthSvc & UserSvc & AdminSvc & LobbySvc & RuntimeSvc & MailSvc & NotifSvc & GeoSvc --> Postgres

    MailWorker --> Postgres
    MailWorker --> SMTP
    NotifWorker --> Postgres
    NotifWorker --> MailSvc & PushSvc
    Sweeper --> LobbySvc
    RuntimeWorkers --> Docker
    RuntimeWorkers --> Game
    RuntimeWorkers --> RuntimeSvc

    GeoSvc --> GeoDB

    HTTP & Push & MailWorker & NotifWorker & Sweeper & RuntimeWorkers --> Telemetry

Process lifecycle

internal/app.App orchestrates startup and shutdown. The start order is fixed:

1. Load configuration with internal/config.LoadFromEnv and validate.
2. Build the zap logger and OpenTelemetry runtime.
3. Open the Postgres pool through internal/postgres.Open.
4. Apply embedded migrations with pressly/goose/v3 before any listener binds.
5. Build the push service (no listener yet) so domain modules can be given a real publisher.
6. Build domain services in dependency order: geo → user (uses geo) → mail → auth (uses user, mail, push) → admin → lobby (uses runtime adapter, notification adapter, user-entitlement adapter) → runtime (uses lobby consumer) → notification (uses mail, push, accounts).
7. Warm every cache (auth, user, admin, lobby, runtime). Each cache exposes Ready(); /readyz waits on every flag.
8. Wire HTTP handlers and the gin engine.
9. Start the HTTP server, the gRPC push server, the mail worker, the notification worker, the lobby sweeper, the runtime worker pool, the runtime scheduler, and the reconciler. The optional Prometheus metrics server is added only when configured.

app.New accepts a shutdownTimeout (BACKEND_SHUTDOWN_TIMEOUT, default 30s). On SIGINT/SIGTERM, components are stopped in reverse order:

1. Refuse new HTTP and gRPC traffic.
2. Drain in-flight requests (BACKEND_HTTP_SHUTDOWN_TIMEOUT, BACKEND_GRPC_PUSH_SHUTDOWN_TIMEOUT).
3. Flush the mail worker's currently-running attempt; pending rows stay in the database for the next process to pick up.
4. Flush push events that already left domain services to the gateway buffer.
5. Drain pending geo counter goroutines.
6. Close the Docker client and the runtime engine HTTP client.
7. Close the Postgres pool.
8. Shut down telemetry, flushing any buffered traces.

The smaller of BACKEND_SHUTDOWN_TIMEOUT and the per-component deadline always wins.

Cyclic dependency adapters

Several domain pairs are mutually dependent (auth↔user for session revoke on permanent block; lobby↔runtime for start/stop calls and snapshot push-back; user/lobby/runtime↔notification for fan-out publishers). The wiring code in cmd/backend/main.go constructs a small adapter struct first, then patches its inner pointer once the real service exists. The adapters live next to the wiring code and never grow domain logic; they are pure forwarders that fall back to a no-op when the inner pointer is still nil (the initial state during boot).

Worker pools

• Mail worker (internal/mail.Worker) — single goroutine that scans mail_deliveries with SELECT ... FOR UPDATE SKIP LOCKED, sends through SMTP, records the attempt, and either marks sent or schedules next_attempt_at with backoff plus jitter. Drains pending and retrying rows on startup.
• Notification worker (internal/notification.Worker) — same pattern over notification_routes: pulls a route, dispatches push or email, writes the outcome, and either marks delivered or moves the route into notification_dead_letters after the configured attempt budget.
• Lobby sweeper (internal/lobby.Sweeper) — pkg/cronutil job that releases pending_registration Race Name Directory entries past BACKEND_LOBBY_PENDING_REGISTRATION_TTL and auto-closes enrollment-expired games whose approved_count >= min_players.
• Runtime worker pool (internal/runtime.Workers) — bounded concurrency (BACKEND_RUNTIME_WORKER_POOL_SIZE) over a buffered channel (BACKEND_RUNTIME_JOB_QUEUE_SIZE). Long-running pulls and starts execute here; the calling path returns as soon as the job is queued. After Docker reports the container running, the worker polls the engine /healthz until the listener is bound (Docker marks a container running as soon as the entrypoint starts; the Go binary inside takes a moment to bind its TCP port). Only after /healthz succeeds does the worker call /admin/init, passing the same game_id the backend uses to mount the engine's storage directory; the engine echoes it back in StateResponse.id. The engine rejects a mismatched gameId with 409 Conflict.
• Runtime scheduler (internal/runtime.SchedulerComponent) — pkg/cronutil schedule per running game; each tick invokes the engine admin/turn. Force-next-turn flips a one-shot skip flag in runtime_records; the next scheduled tick observes the flag and consumes it.
• Runtime reconciler (internal/runtime.Reconciler) — periodic list of containers labelled galaxy.backend=1, matched against runtime_records. Adopts unrecorded labelled containers, marks recorded but missing as removed, and emits lobby.OnRuntimeJobResult for the latter.

Telemetry

Tracing covers HTTP request → domain operation → Postgres call → external client (SMTP, Docker, engine). zap injects otel_trace_id and otel_span_id into every log entry written inside a request scope. OTel exporters honour BACKEND_OTEL_TRACES_EXPORTER and BACKEND_OTEL_METRICS_EXPORTER; both default to otlp and accept none, stdout, and (for metrics) prometheus.

TraceFieldsFromContext(ctx) is exposed by internal/telemetry.Runtime rather than the logger package because the helper is used by middleware and depends on the OTel runtime, not the logger configuration. Keeping it next to the runtime keeps server → telemetry import direction one-way.