refactor(dev): remove the dev-sandbox bootstrap everywhere
Tests · Go / test (push) Successful in 1m59s
Tests · Go / test (push) Successful in 1m59s
Stage 1 of the dev-as-prod-mirror rework. The auto-provisioned "Dev Sandbox" game and dummy users are removed so the dev contour starts empty like prod; the separate legacy-report loader stays as the test-data path. - delete backend/internal/devsandbox (package + tests) - drop the bootstrap call + DevSandboxConfig (struct, Config field, BACKEND_DEV_SANDBOX_* env, defaults, loader, validation) - strip BACKEND_DEV_SANDBOX_* from dev-deploy + local-dev compose and .env.example; the generic engine-recycle / prune-broken-engines logic stays (it serves real games) - update tooling docs (dev-deploy README + KNOWN-ISSUES, local-dev README + Makefile) and stale comments; DeleteGame and InsertMembershipDirect remain (exercised by lobby integration tests) No app behaviour change beyond not auto-creating the sandbox game.
This commit is contained in:
Ilia Denisov
@@ -7,12 +7,6 @@
|
||||
# baked into `docker-compose.yml`, so this file documents the knobs
|
||||
# rather than driving them.
|
||||
|
||||
# Auto-provisioned sandbox bootstrap. Empty disables the bootstrap.
|
||||
BACKEND_DEV_SANDBOX_EMAIL=dev@galaxy.lan
|
||||
BACKEND_DEV_SANDBOX_ENGINE_IMAGE=galaxy-engine:dev
|
||||
BACKEND_DEV_SANDBOX_ENGINE_VERSION=0.1.0
|
||||
BACKEND_DEV_SANDBOX_PLAYER_COUNT=20
|
||||
|
||||
# `123456` short-circuits the email-code path for the dev account.
|
||||
# This is also the docker-compose default — set the variable to an
|
||||
# empty string here when the environment must rely on real Mailpit
|
||||
|
||||
@@ -1,164 +1,8 @@
|
||||
# `tools/dev-deploy/` — known issues
|
||||
|
||||
Issues that surface in the long-lived dev environment but are not yet
|
||||
fixed. Each entry lists the observed symptom, the diagnostic evidence,
|
||||
the working hypothesis, and the open questions that have to be
|
||||
answered before a fix lands.
|
||||
|
||||
## Dev Sandbox game flips to `cancelled` after a `dev-deploy` redispatch
|
||||
|
||||
### Symptom
|
||||
|
||||
A previously `running` "Dev Sandbox" game (created by
|
||||
`backend/internal/devsandbox`) transitions to `cancelled` ~15 minutes
|
||||
after a `dev-deploy.yaml` workflow_dispatch run finishes. The user's
|
||||
browser session survives (the same `device_session_id` keeps working),
|
||||
but the lobby shows no game because the only game it had is now
|
||||
terminal. `purgeTerminalSandboxGames` does pick it up on the **next**
|
||||
boot and creates a fresh sandbox — but the first redispatch leaves
|
||||
the user with an empty lobby until backend restarts again.
|
||||
|
||||
### Diagnostic evidence
|
||||
|
||||
Backend logs from the broken cycle (timestamps abbreviated):
|
||||
|
||||
```text
|
||||
20:24:40 dev_sandbox: purged terminal sandbox game game_id=<prev> status=cancelled
|
||||
20:24:40 dev_sandbox: memberships ensured count=20 game_id=<new>
|
||||
20:24:40 dev_sandbox: bootstrap complete user_id=<owner> game_id=<new> status=starting
|
||||
...
|
||||
20:25:09 user mail sent failed (diplomail tables missing — unrelated)
|
||||
...
|
||||
20:39:40 lobby: game cancelled by runtime reconciler game_id=<new>
|
||||
op=reconcile status=removed message="container disappeared"
|
||||
```
|
||||
|
||||
Between 20:24:40 (`status=starting`) and 20:39:40 (reconciler cancel)
|
||||
the backend logs are silent on the runtime / engine paths — no
|
||||
`engine spawned`, no `engine container started`, no `runtime
|
||||
transition` lines. The reconciler then fires and reports the engine
|
||||
container as missing.
|
||||
|
||||
`docker ps -a --filter 'label=org.opencontainers.image.title=galaxy-game-engine'`
|
||||
returns no rows during this window — the engine container is neither
|
||||
running nor stopped on the host, so it either was never spawned or
|
||||
was removed before the host snapshot.
|
||||
|
||||
### What has been ruled out
|
||||
|
||||
A live `docker inspect` on a healthy engine container shows:
|
||||
|
||||
```text
|
||||
Labels: galaxy.backend=1, galaxy.engine_version=0.1.0,
|
||||
galaxy.game_id=<uuid>,
|
||||
org.opencontainers.image.title=galaxy-game-engine,
|
||||
com.galaxy.{cpu_quota,memory,pids_limit}
|
||||
AutoRemove: false
|
||||
RestartPolicy: on-failure
|
||||
NetworkMode: galaxy-dev-internal
|
||||
```
|
||||
|
||||
There are no `com.docker.compose.*` labels and `AutoRemove=false`,
|
||||
so `--remove-orphans` cannot reap the engine and a `--rm`-style
|
||||
self-destruct is not in play. Two redispatches captured under
|
||||
`docker events --filter event=create,start,die,destroy,kill,stop`
|
||||
also confirmed it: across both runs the only `die` / `destroy`
|
||||
events were for `galaxy-dev-{backend,api,caddy}`. The live engine
|
||||
container survived both redispatches, and the reconciler that
|
||||
fires 60 seconds after the new backend boots correctly matched
|
||||
it through `byGameID` / `byContainerID`.
|
||||
|
||||
`backend/internal/runtime/service.go` only removes engine
|
||||
containers from the explicit `runStop` / `runRestart` / `runPatch`
|
||||
paths. There is no `runtime.Service.Shutdown` that proactively
|
||||
kills containers on backend exit, so a graceful SIGTERM to
|
||||
`galaxy-dev-backend` will not touch its child engine containers.
|
||||
|
||||
### Host-side hypotheses considered and rejected by the owner
|
||||
|
||||
The natural follow-up suspects after compose was cleared — host-side
|
||||
`docker prune` cron jobs, a manual `docker rm`, an out-of-band
|
||||
`dockerd` restart, and an idle-state engine crash — were all
|
||||
rejected by the project owner: the dev host runs none of those
|
||||
periodic cleanups, no one manually removed the container, dockerd
|
||||
was not restarted in the window, and the engine binary does not
|
||||
crash while idling on API calls.
|
||||
|
||||
### Best remaining suspicion
|
||||
|
||||
Something the `dev-deploy.yaml` CI run does between successful
|
||||
image builds and the final `docker compose up -d --wait
|
||||
--remove-orphans` clobbers the previously-spawned engine container.
|
||||
The chain at runtime contains:
|
||||
|
||||
1. `docker build -t galaxy-engine:dev -f game/Dockerfile .`
|
||||
2. `docker compose build galaxy-backend galaxy-api`
|
||||
3. `docker run --rm` alpine for the UI volume seed
|
||||
4. `docker compose up -d --wait --remove-orphans`
|
||||
|
||||
None of these *should* touch an unmanaged engine container, but
|
||||
the reproduction window points squarely inside this sequence. A
|
||||
deliberate next reproduction with `docker events --since 0` armed
|
||||
*before* the deploy starts and live for the entire job — captured
|
||||
end-to-end on the dev host, not just the chunk after backend
|
||||
recreate — would pin which step emits the `destroy` on the engine.
|
||||
|
||||
### Update 2026-05-19: integration preclean identified as one cause
|
||||
|
||||
A live reproduction during the post-merge auto-deploy cycle (Gitea
|
||||
run #188 dev-deploy plus parallel run #190 integration) pinned one
|
||||
clobbering source: `integration/scripts/preclean.sh` was unscoped
|
||||
and removed *every* container labelled `galaxy.backend=1`, including
|
||||
the dev-deploy engine. Timeline from the dev host:
|
||||
|
||||
```text
|
||||
23:10:40 backend pre-bootstrap reconciler tick: engine alive
|
||||
23:10:40 dev_sandbox bootstrap: status=running
|
||||
23:10:56 preclean: removing 1 backend-managed engine containers ← integration run #190
|
||||
23:11:40 reconciler: container disappeared → game cancelled
|
||||
```
|
||||
|
||||
Fix landed: `BACKEND_STACK_LABEL=integration` is now passed to
|
||||
every integration backend (see
|
||||
`integration/testenv/backend.go`) and `preclean.sh` AND-combines
|
||||
`galaxy.backend=1` with `galaxy.stack=integration`, so dev-deploy /
|
||||
local-dev engines stamped with different stack values are no longer
|
||||
collateral.
|
||||
|
||||
This covers **push**-triggered cycles where `dev-deploy.yaml` and
|
||||
`integration.yaml` run on the same Gitea host. The original
|
||||
hypothesis (a `workflow_dispatch dev-deploy` solo run also losing
|
||||
the engine) is *not* explained by the integration fix — manual
|
||||
dispatches do not trigger `integration.yaml`. Keep this entry open
|
||||
until a solo-dispatch reproduction confirms whether the symptom
|
||||
still occurs.
|
||||
|
||||
### Status
|
||||
|
||||
Partially fixed (push-triggered cycles). Solo `workflow_dispatch`
|
||||
reproductions still open. If the symptom recurs after the
|
||||
integration fix lands, capture `docker events --since 0` for the
|
||||
full dispatch window and attach here.
|
||||
|
||||
### Workaround in use today
|
||||
|
||||
When the sandbox game flips to `cancelled`, redispatch `dev-deploy`:
|
||||
|
||||
```sh
|
||||
curl -X POST -n -H 'Content-Type: application/json' \
|
||||
-d '{"ref":"<branch>"}' \
|
||||
https://gitea.iliadenisov.ru/api/v1/repos/developer/galaxy-game/actions/workflows/dev-deploy.yaml/dispatches
|
||||
```
|
||||
|
||||
The next boot's `purgeTerminalSandboxGames` removes the cancelled
|
||||
row, `findOrCreateSandboxGame` creates a fresh one, and
|
||||
`ensureMembershipsAndDrive` puts the new game back to `running`.
|
||||
|
||||
### Owner
|
||||
|
||||
Unassigned. File an issue once we have the runtime / reconciler
|
||||
analysis above; reference this section in the issue body so future
|
||||
redeploys can short-circuit the diagnostic loop.
|
||||
Issues that surfaced in the long-lived dev environment. Each entry lists
|
||||
the observed symptom, the diagnostic evidence, and the fix or the open
|
||||
questions that have to be answered before a fix lands.
|
||||
|
||||
## `docker restart galaxy-dev-backend` fails after the CI runner cleans up
|
||||
|
||||
|
||||
@@ -114,8 +114,7 @@ calls `make clean-data`.
|
||||
The same dev-mode email-code override as `tools/local-dev/` applies,
|
||||
and the dev-deploy compose ships with it enabled by default:
|
||||
|
||||
1. Enter `dev@galaxy.lan` (or whatever `BACKEND_DEV_SANDBOX_EMAIL`
|
||||
resolves to) in the login form.
|
||||
1. Enter your email address in the login form.
|
||||
2. Submit `123456` as the code — the docker-compose default for
|
||||
`BACKEND_AUTH_DEV_FIXED_CODE` is `123456`, so the bcrypt-hashed
|
||||
email code stays a fallback. To force real Mailpit codes (e.g. for
|
||||
@@ -212,8 +211,7 @@ make clean-data Stop everything and wipe volumes + game-state dir
|
||||
## Known issues
|
||||
|
||||
See [`KNOWN-ISSUES.md`](KNOWN-ISSUES.md) for symptoms that surface
|
||||
in the long-lived dev environment but are not yet fixed (currently:
|
||||
the sandbox game flipping to `cancelled` after a redispatch).
|
||||
in the long-lived dev environment but are not yet fixed.
|
||||
|
||||
## Deployment cadence
|
||||
|
||||
@@ -237,12 +235,12 @@ behind. There is no separate state to clean up between the two paths.
|
||||
|
||||
### Engine image drift recycle
|
||||
|
||||
`backend` spawns one engine container per game (the long-lived "Dev
|
||||
Sandbox" plus any user-created games) and the reconciler reattaches
|
||||
to whatever it finds with the `galaxy.stack=dev-deploy` label. That
|
||||
reattach does not check the running container's image SHA against the
|
||||
freshly-built `galaxy-engine:dev` tag, so an unchanged container would
|
||||
otherwise keep serving the previous engine code after a redeploy.
|
||||
`backend` spawns one engine container per running game and the
|
||||
reconciler reattaches to whatever it finds with the
|
||||
`galaxy.stack=dev-deploy` label. That reattach does not check the
|
||||
running container's image SHA against the freshly-built
|
||||
`galaxy-engine:dev` tag, so an unchanged container would otherwise
|
||||
keep serving the previous engine code after a redeploy.
|
||||
|
||||
The `dev-deploy.yaml` workflow handles this in the
|
||||
`Recycle engine containers on image drift` step. When `docker build`
|
||||
@@ -250,9 +248,7 @@ produces a new `galaxy-engine:dev` SHA, the step compares it against
|
||||
every running `galaxy-game-*` container and, for each drifted one,
|
||||
stops the backend, removes the container, wipes its bind-mounted
|
||||
state directory (Engine.Init() writes turn-0 over any pre-existing
|
||||
`turn-N` files), and cascade-deletes the lobby `games` row. The
|
||||
`dev-sandbox` bootstrap on the next backend boot finds no live
|
||||
sandbox and provisions a fresh one on the new engine image.
|
||||
`turn-N` files), and cascade-deletes the lobby `games` row.
|
||||
|
||||
When the engine sources are unchanged, the BuildKit cache hits and
|
||||
the SHA stays the same — the recycle step is a no-op and the running
|
||||
|
||||
@@ -127,15 +127,6 @@ services:
|
||||
# bcrypt-hashed code is single-use). Set the var to an empty
|
||||
# string in `.env` to disable.
|
||||
BACKEND_AUTH_DEV_FIXED_CODE: ${BACKEND_AUTH_DEV_FIXED_CODE:-123456}
|
||||
# Long-lived dev environment always bootstraps the "Dev Sandbox"
|
||||
# game owned by this email so a freshly redeployed stack already
|
||||
# has one ready-to-play game in the lobby. Set the variable to an
|
||||
# empty string in `.env` to disable the bootstrap (e.g. for a
|
||||
# cold-start QA pass).
|
||||
BACKEND_DEV_SANDBOX_EMAIL: ${BACKEND_DEV_SANDBOX_EMAIL:-dev@galaxy.lan}
|
||||
BACKEND_DEV_SANDBOX_ENGINE_IMAGE: ${BACKEND_DEV_SANDBOX_ENGINE_IMAGE:-galaxy-engine:dev}
|
||||
BACKEND_DEV_SANDBOX_ENGINE_VERSION: ${BACKEND_DEV_SANDBOX_ENGINE_VERSION:-0.1.0}
|
||||
BACKEND_DEV_SANDBOX_PLAYER_COUNT: ${BACKEND_DEV_SANDBOX_PLAYER_COUNT:-20}
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
# Per-game state directories live under the same absolute path
|
||||
|
||||
Reference in New Issue
Block a user