# `tools/dev-deploy/` — known issues Issues that surface in the long-lived dev environment but are not yet fixed. Each entry lists the observed symptom, the diagnostic evidence, the working hypothesis, and the open questions that have to be answered before a fix lands. ## Dev Sandbox game flips to `cancelled` after a `dev-deploy` redispatch ### Symptom A previously `running` "Dev Sandbox" game (created by `backend/internal/devsandbox`) transitions to `cancelled` ~15 minutes after a `dev-deploy.yaml` workflow_dispatch run finishes. The user's browser session survives (the same `device_session_id` keeps working), but the lobby shows no game because the only game it had is now terminal. `purgeTerminalSandboxGames` does pick it up on the **next** boot and creates a fresh sandbox — but the first redispatch leaves the user with an empty lobby until backend restarts again. ### Diagnostic evidence Backend logs from the broken cycle (timestamps abbreviated): ```text 20:24:40 dev_sandbox: purged terminal sandbox game game_id= status=cancelled 20:24:40 dev_sandbox: memberships ensured count=20 game_id= 20:24:40 dev_sandbox: bootstrap complete user_id= game_id= status=starting ... 20:25:09 user mail sent failed (diplomail tables missing — unrelated) ... 20:39:40 lobby: game cancelled by runtime reconciler game_id= op=reconcile status=removed message="container disappeared" ``` Between 20:24:40 (`status=starting`) and 20:39:40 (reconciler cancel) the backend logs are silent on the runtime / engine paths — no `engine spawned`, no `engine container started`, no `runtime transition` lines. The reconciler then fires and reports the engine container as missing. `docker ps -a --filter 'label=org.opencontainers.image.title=galaxy-game-engine'` returns no rows during this window — the engine container is neither running nor stopped on the host, so it either was never spawned or was removed before the host snapshot. ### What has been ruled out A live `docker inspect` on a healthy engine container shows: ```text Labels: galaxy.backend=1, galaxy.engine_version=0.1.0, galaxy.game_id=, org.opencontainers.image.title=galaxy-game-engine, com.galaxy.{cpu_quota,memory,pids_limit} AutoRemove: false RestartPolicy: on-failure NetworkMode: galaxy-dev-internal ``` There are no `com.docker.compose.*` labels and `AutoRemove=false`, so `--remove-orphans` cannot reap the engine and a `--rm`-style self-destruct is not in play. Two redispatches captured under `docker events --filter event=create,start,die,destroy,kill,stop` also confirmed it: across both runs the only `die` / `destroy` events were for `galaxy-dev-{backend,api,caddy}`. The live engine container survived both redispatches, and the reconciler that fires 60 seconds after the new backend boots correctly matched it through `byGameID` / `byContainerID`. `backend/internal/runtime/service.go` only removes engine containers from the explicit `runStop` / `runRestart` / `runPatch` paths. There is no `runtime.Service.Shutdown` that proactively kills containers on backend exit, so a graceful SIGTERM to `galaxy-dev-backend` will not touch its child engine containers. ### Host-side hypotheses considered and rejected by the owner The natural follow-up suspects after compose was cleared — host-side `docker prune` cron jobs, a manual `docker rm`, an out-of-band `dockerd` restart, and an idle-state engine crash — were all rejected by the project owner: the dev host runs none of those periodic cleanups, no one manually removed the container, dockerd was not restarted in the window, and the engine binary does not crash while idling on API calls. ### Best remaining suspicion Something the `dev-deploy.yaml` CI run does between successful image builds and the final `docker compose up -d --wait --remove-orphans` clobbers the previously-spawned engine container. The chain at runtime contains: 1. `docker build -t galaxy-engine:dev -f game/Dockerfile .` 2. `docker compose build galaxy-backend galaxy-api` 3. `docker run --rm` alpine for the UI volume seed 4. `docker compose up -d --wait --remove-orphans` None of these *should* touch an unmanaged engine container, but the reproduction window points squarely inside this sequence. A deliberate next reproduction with `docker events --since 0` armed *before* the deploy starts and live for the entire job — captured end-to-end on the dev host, not just the chunk after backend recreate — would pin which step emits the `destroy` on the engine. ### Update 2026-05-19: integration preclean identified as one cause A live reproduction during the post-merge auto-deploy cycle (Gitea run #188 dev-deploy plus parallel run #190 integration) pinned one clobbering source: `integration/scripts/preclean.sh` was unscoped and removed *every* container labelled `galaxy.backend=1`, including the dev-deploy engine. Timeline from the dev host: ```text 23:10:40 backend pre-bootstrap reconciler tick: engine alive 23:10:40 dev_sandbox bootstrap: status=running 23:10:56 preclean: removing 1 backend-managed engine containers ← integration run #190 23:11:40 reconciler: container disappeared → game cancelled ``` Fix landed: `BACKEND_STACK_LABEL=integration` is now passed to every integration backend (see `integration/testenv/backend.go`) and `preclean.sh` AND-combines `galaxy.backend=1` with `galaxy.stack=integration`, so dev-deploy / local-dev engines stamped with different stack values are no longer collateral. This covers **push**-triggered cycles where `dev-deploy.yaml` and `integration.yaml` run on the same Gitea host. The original hypothesis (a `workflow_dispatch dev-deploy` solo run also losing the engine) is *not* explained by the integration fix — manual dispatches do not trigger `integration.yaml`. Keep this entry open until a solo-dispatch reproduction confirms whether the symptom still occurs. ### Status Partially fixed (push-triggered cycles). Solo `workflow_dispatch` reproductions still open. If the symptom recurs after the integration fix lands, capture `docker events --since 0` for the full dispatch window and attach here. ### Workaround in use today When the sandbox game flips to `cancelled`, redispatch `dev-deploy`: ```sh curl -X POST -n -H 'Content-Type: application/json' \ -d '{"ref":""}' \ https://gitea.iliadenisov.ru/api/v1/repos/developer/galaxy-game/actions/workflows/dev-deploy.yaml/dispatches ``` The next boot's `purgeTerminalSandboxGames` removes the cancelled row, `findOrCreateSandboxGame` creates a fresh one, and `ensureMembershipsAndDrive` puts the new game back to `running`. ### Owner Unassigned. File an issue once we have the runtime / reconciler analysis above; reference this section in the issue body so future redeploys can short-circuit the diagnostic loop. ## `docker restart galaxy-dev-backend` fails after the CI runner cleans up **Status: fixed (2026-05-19).** Kept here as a postmortem in case the symptom resurfaces in a different form. ### Symptom `docker restart galaxy-dev-backend` from the host failed with: ```text Error response from daemon: ... error mounting "/home/runner/.cache/act//hostexecutor/pkg/geoip/test-data/test-data/GeoIP2-Country-Test.mmdb" to rootfs at "/var/lib/galaxy/geoip.mmdb": ... not a directory ``` The container ended up `Exited (127)` and never came back. ### Cause `tools/dev-deploy/docker-compose.yml` used to mount the geoip database via a path relative to the compose file (`../../pkg/geoip/test-data/test-data/GeoIP2-Country-Test.mmdb`). When the `dev-deploy.yaml` Gitea runner invoked `docker compose up`, it resolved that relative path against the runner's ephemeral workspace under `/home/runner/.cache/act//hostexecutor/tools/dev-deploy/`, so the bind-mount source baked into the running container pointed at that ephemeral path. The runner deleted the workspace once the workflow ended, the source disappeared, and the next `docker restart` failed to remount it. ### Fix Replaced the bind-mount with a named volume, `galaxy-dev-geoip-data`, seeded by the `dev-deploy.yaml` workflow (and by the new `make seed-geoip` target) at deploy time. The backend mounts the volume as `/var/lib/galaxy:ro`, so the bind source is a Docker-managed volume — independent of the runner workspace — and survives a `docker restart`. See `.gitea/workflows/dev-deploy.yaml` ("Seed geoip volume" step) and `tools/dev-deploy/Makefile` (`seed-geoip` target).