The gateway and bot run on distroless nonroot (UID 65532) and bind-mount the
cert dir read-only, but gen-certs.sh wrote the keys 0600 (owner-only, the deploy
user), so both crash-looped at boot with "open /certs/*.key: permission denied"
and the deploy probe correctly failed (the contour's gateway was down).
The .crt files were already 0644 (openssl default); make the leaf keys 0644 too
so UID 65532 can read them. These are ephemeral TEST certificates regenerated
every deploy on the trusted runner; prod keys come from PROD_ secrets. The CA
private key stays 0600 (containers never read it).
Move all Telegram egress off the main host. The single connector held the
bot token, long-polled Telegram and answered the gateway/backend over the
trusted internal network, so the whole component (including login validation)
shared fate with its VPN sidecar. Split it into two binaries that share the
token:
- cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only,
never calls the Bot API. The gateway dials it for Telegram auth, so game
login is now independent of Telegram reachability.
- cmd/bot (remote): Bot API long-poll + sendMessage, the only component
reaching Telegram. It holds no inbound port — it dials the gateway over a
new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send
commands the gateway pushes.
The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget
(at-most-once, dropped if no bot is connected); the backend admin broadcasts
reach a gateway-served relay that forwards them and awaits the bot's ack
(SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one
inter-service link that leaves the trusted segment; validator<->gateway and
the relay stay plaintext internal. The bot is Telegram-rate-limited.
One bot now; the gateway bot registry, an owns_updates flag and per-command
ids leave seams for N later. Webhook rejected (one URL per token, adds inbound
+ a static address).
The unified test contour runs the split (the bot keeps its VPN sidecar and
dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh,
generated in CI). The prod wiring — the bot on a separate host (no VPN), the
gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH
deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX,
Stage 18).
Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway +
backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
The seed-drift guard shipped as refuse-boot: the backend exited when
BACKEND_DICT_VERSION disagreed with the flat dir's recorded .seed_version. On
the test contour that turned a harmless-in-intent action — bumping the
TEST_DICT_VERSION variable to the active release (v1.2.1) on a volume seeded as
v1.0.0 — into a crash loop, because DICT_VERSION is the *seed* of a fresh
volume, not the active version (which the admin console drives).
Make the marker authoritative instead: OpenWithVersions resolves the flat dir's
version from .seed_version when present and ignores bootVersion on an
already-seeded volume; bootVersion only seeds a fresh volume's marker. So a
bumped build seed on a live volume is a no-op (it can't relabel live bytes and
can't void games pinned to the prior label), and it correctly seeds the next
fresh volume. The subdirectory scan now skips the resolved seed, so a version
also present as a subdir (e.g. v1.2.1 uploaded via the console while the build
seed is bumped to v1.2.1) is still loaded rather than shadowed by the flat bytes.
Tests: marker-wins over a bumped boot version; a bumped boot keeps the matching
subdir resident (the live-contour case). Docs updated (ARCHITECTURE §5, READMEs,
compose/.env, PRERELEASE DV) from "refuses to boot" to "marker wins / ignored".
Verified locally against v1.2.1: gofmt, build, vet, unit, integration green.
The dictionary release moved to v1.2.1 while DICT_VERSION stayed pinned at
v1.0.0 in CI and the image/compose seed defaults. Two problems:
1. CI validated against a stale dictionary.
2. The contour seed could be bumped on a live volume, which silently relabels
the already-seeded bytes — voiding games pinned to the prior label and
serving the wrong dictionary for new ones. The flat DAWGs carry no embedded
version, so this drift was undetectable.
Changes:
- Seed-drift guard: OpenWithVersions records the flat dir's version in a
.seed_version marker on first boot and refuses to start when a later
BACKEND_DICT_VERSION disagrees. DICT_VERSION is now the seed for a *fresh*
volume only; a live contour migrates through the admin console (old versions
stay resident, in-progress games keep replaying).
- Track the current release: CI's DICT_VERSION centralised to one workflow-level
env (v1.2.1); image/compose/.env seed defaults bumped to v1.2.1. The deploy
job keeps reading the per-contour vars.TEST_DICT_VERSION.
- Docs: ARCHITECTURE §5 (decision record), backend/deploy READMEs, PRERELEASE
tracker (DV row).
Verified locally against the v1.2.1 artifact: gofmt, build, vet, unit and
integration (-tags=integration) all green.
Collapse the two per-language Telegram bots into one unified bot and
replace language-based variant gating with explicit per-user variant
preferences.
- Telegram: one bot; drop service_language and the supported_languages
set everywhere (DB, account, auth, FlatBuffers Session wire, gateway,
connector proto). The single bot renders chat and out-of-app push in
the recipient's preferred_language; remove the game-language push
routing override (notify Intent.Language / push Event.language).
- Preferences: new accounts.variant_preferences (text[], DB default
{erudit_ru}, CHECK non-empty + subset of the three variants). Gates
the New Game picker, vs-AI and the friend invitation the player
creates, enforced server-side (HTTP 400 otherwise); an invited friend
may still accept any variant. Edited on the Settings screen; variants
are Erudit-first everywhere.
- Admin: drop the per-bot language selectors (broadcast / send-to-user)
and the feedback channel_lang column/field.
- Env/CI: collapse TELEGRAM_BOT_TOKEN_{EN,RU}, TELEGRAM_GAME_CHANNEL_ID_{EN,RU},
VITE_TELEGRAM_LINK{,_EN,_RU} and VITE_TELEGRAM_GAME_CHANNEL_NAME_{EN,RU}
to single unsuffixed names; drop GATEWAY_DEFAULT_SUPPORTED_LANGUAGES.
- Docs updated (ARCHITECTURE, FUNCTIONAL + _ru, platform/telegram, gateway,
backend, ui, UI_DESIGN, PRERELEASE).
The migration squash is deferred to a follow-up PR.
Visualises the chat read-receipt metrics added with the read-receipt feature
(the dashboard was deferred there per the owner): posted rate by kind, the
unread backlog (chat_unread_messages gauge), and the publish-to-read latency
(chat_read_duration — average by kind plus overall p50/p95). Mirrors the
game-domain dashboard's structure; the file provider auto-discovers it.
Follow-ups on the honest-AI game, same PR:
- GCG export labels the robot seat "AI" instead of its pool name (ExportGCG
overrides via accounts.IsRobot); the in-app 🤖 is unchanged.
- vs_ai games emit no your_turn (the robot replies instantly, so it would be
redundant); opponent_moved still advances the UI.
- Admin console shows the AI flag: a 🤖 column in /games and an "AI game" line
on the game card (GameRow/GameDetailView gain VsAI).
- games_started_total / games_abandoned_total gain a vs_ai attribute; the
Grafana Game-domain dashboard splits started/abandoned into human and AI
panels.
Tests: metrics unit (vs_ai split); integration (no your_turn, GCG "AI").
The friend-code 'share' was an <a> that just opened the bot. Make it a real share:
Telegram's native share-to-chat picker inside the Mini App (openTelegramLink +
t.me/share/url), the system share sheet (navigator.share) on the web, else copy the
link. The shared deep link points at the same bot the player is in — it picks
VITE_TELEGRAM_LINK_EN/_RU by the session's service language, falling back to the
single VITE_TELEGRAM_LINK. Adds the per-bot build args across Dockerfile / compose /
ci.yaml / .env / docs; PLAN TODO-5 updated.
Replace the dictionary hot-reload with an online update flow in the GM console
(/_gm/dictionary): the operator uploads scrabble-dawg-vX.Y.Z.tar.gz, previews the
per-variant words added/removed against the active dictionary, and confirms to
install + activate it. Versions are immutable; in-progress games keep their pinned
version while new games use the new one.
- engine: DiffWords (enumerate both DAWGs, decode only the differences), OpenFinder,
Registry.Finder, DictFiles; OpenWithVersions skips the .staging area.
- dictadmin: hardened release-archive validation + extraction (path-traversal,
symlink, oversize, entry-count rejection) and staging -> install (atomic rename).
- game: active dictionary version persisted in the dictionary_state singleton
(single source of truth, restored on boot), concurrency-safe accessor.
- storage: BACKEND_DICT_DIR is a named volume seeded from the image (nonroot-owned),
so uploaded versions persist across redeploys; the build's DICT_VERSION labels the
seed and equals the resident tag (BACKEND_DICT_VERSION).
- docs: ARCHITECTURE §5, FUNCTIONAL (+ru), backend README, TESTING, PRERELEASE.
Tests: engine + dictadmin unit; integration upload->preview->install->activate->
restart->pin->immutability->CSRF.
Round-2 tuning, decided from the 500-player resource profile:
- gateway: 2 -> 3 cores + GOMAXPROCS=3. It holds one h2c connection per player, so
at 500 players it burst into the 2-core cap (~2.49% transport_error on game.state);
3 cores absorbs the bursts. The per-connection cost is the realistic prod load.
- tempo: memory 1G -> 2G. It reached the 1 GiB cap during the run (OOM risk).
- backend Postgres pool: MAX_OPEN_CONNS 25 -> 40. The pool sat at its 25-conn cap
(28 backends) at peak; headroom trims the p99 tail. Postgres (2c/512M) handles it.
- docker log volume: a json-file rotation default (10m x 3 = 30 MiB/container) applied
contour-wide via a YAML anchor; the backend logs ~14 MiB / 30 min at info under load
and was previously unbounded. Log level stays info.
backend/postgres stay at 2 cores / 512 MiB (peak ~0.85 / ~1.4 cores — headroom is cheap
on the shared host). A validation re-run confirms the gateway fix before merge.
The receiver defaults to Docker API 1.25, but the contour daemon's minimum is
1.40 (it speaks up to 1.54), so otelcol crash-looped on start with "client
version 1.25 is too old". Pinning api_version to 1.44 (accepted by both the
receiver's bundled client and the daemon) starts the receiver cleanly —
verified by running the image against the host socket ("Everything is ready",
no start error).
Observability: replace cAdvisor (which resolves only the root cgroup on the
contour host — separate-XFS /var/lib/docker) with the otelcol docker_stats
receiver, which reads per-container CPU/memory/network straight from the Docker
API and works the same in prod. The collector joins the host docker group
(DOCKER_GID, default 989) and mounts the socket read-only; its metrics flow out
through the existing prometheus exporter, so the cAdvisor scrape job and the
privileged cAdvisor service are removed. The Resources dashboard panels are
retargeted to the docker_stats metric names (container_name label;
container.cpu.utilization/100 == cores).
Container limits: apply deploy.resources.limits (honoured by Compose v2) across
the contour and pin GOMAXPROCS to the CPU limit on the Go services so the runtime
matches the cgroup quota. Starting values are generous over the R2 peak (~1 core /
<=100 MiB per app service) to avoid skewing or OOM-killing the measurement run;
they are tightened to the agreed prod sizing after the final stress run (R7
Round 2). The privileged VPN sidecar is left unconstrained.
- gateway/Dockerfile gains a `landing` target: caddy:2-alpine + the shared
Vite build (identical build args keep the ui stage a single cached build);
the gateway target drops landing.html from the embed.
- The contour caddy routes /app/, /telegram/ and the Connect path to the
gateway; the catch-all — the landing at / and any stray path — goes to the
new landing service, so junk traffic is absorbed by static file serving.
- deploy/landing/Caddyfile mirrors the webui caching (immutable assets,
no-cache shells) and falls back unknown paths to the landing shell.
- The gateway's / now 308-redirects to /app/ (keeps a local no-caddy run
usable); webui placeholder landing.html removed.
- CI deploy probe checks both / (landing) and /app/ (gateway).
Verified: both images build; the landing container serves landing.html at /
(no-cache) with junk-path fallback; the gateway image redirects / to /app/
and carries no landing content.
New scrabble/loadtest module (the pre-release stress harness): seeds 1000 guest +
10000 durable accounts with pre-created sessions directly in Postgres (token hash
matches backend/internal/session), drives virtual players through the edge protocol
(real 2-4p games assembled via invitations, mid-ranked legal moves generated locally
by the embedded scrabble-solver — the edge carries no board, so the client replays
history), plus nudge/chat/check-word/draft/profile/stats and a gateway-hammer that
verifies the rate limiter. Prints a trip-report summary (per-op latency percentiles,
result codes, live-event tally). Go unit tests cover the pure pieces; the DAWG-backed
move test runs under BACKEND_DICT_DIR.
Contour: add cAdvisor + postgres_exporter + a 'Scrabble - Resources' Grafana
dashboard and the two Prometheus scrape jobs, for the R2/R7 stress-run resource
baseline.
CI: gate ./loadtest/... (path filter + vet/build/test). Docs: TESTING, ARCHITECTURE,
project CLAUDE repo layout.
- Client IP: the compose caddy trusts X-Forwarded-For from private-range
upstreams (trusted_proxies private_ranges), so the real client IP survives
the host-caddy hop (it was logging the docker caddy hop 172.18.0.x for chat
moderation and bucketing the gateway per-IP rate limiter on it). Correct and
spoof-safe in both contours (prod has no host caddy); peerIP unit-tested.
- Ad banner gated off behind a compile-time SHOW_AD_BANNER=false (the if-branch,
the AdBanner import and banner.ts are tree-shaken out of the prod bundle).
- Landing: the Telegram entry is just the 64px logo (clickable, no button/text).
- TG-fullscreen header: title + menu centred as a pair (hamburger right of the
title), pinned to the bottom of the TG nav band.
- Edge-swipe back (Screen): a left-edge rightward drag navigates to back
(touch/pen only, armed from <=24px; skipped inside Telegram).
- Chat soft-keyboard: a bottom-sheet Modal lifted above the keyboard by a
visualViewport-driven transform (compositor-only, no page/sheet relayout).
iOS-specific, needs on-device tuning; native resize=none awaits Capacitor.
- Tests: e2e for the in-game '✓ in friends' item and a board→board tile
relocation; codec units for last_activity_unix + OutgoingRequestList.
Deferred to the next PR (agreed): #4 enrich the your-turn/game-end push; #5 hide
finished games from the lobby.
Owner review-pass rework of the landing page:
- Rename the per-language Telegram link build var
VITE_TELEGRAM_LINK_EN/_RU -> VITE_TELEGRAM_GAME_CHANNEL_NAME_EN/_RU
(it carries a channel username; the landing builds https://t.me/<name> --
the same channels the connector posts to via TELEGRAM_GAME_CHANNEL_ID_*).
- Language switcher -> a globe icon dropdown (flags + names), saved + synced
to the app prefs.
- Theme switcher -> a sun/moon icon toggle, ephemeral (follows the system
scheme, no auto, never persisted) -- galaxy-game style.
- Drop the "Play in browser" CTA (no standalone-web onboarding yet).
Docs: FUNCTIONAL(+ru), PLAN, deploy + ui READMEs.
Close out Stage 17 round 6:
- Landing page at / — one Vite build with two entries (index.html = game
SPA, landing.html = a lightweight landing reusing the theme/i18n/
aboutContent leaf modules, not the app store).
- Move the web game SPA to /app/; the Telegram Mini App stays at /telegram/
(gateway webui.Handler(stripPrefix, indexName): landing at /, SPA at /app/
+ /telegram/). Per-language "Play in Telegram" link via new
VITE_TELEGRAM_LINK_EN/_RU build vars (button hides when unset).
- Cache headers: hash-named /assets/* immutable, HTML shells no-cache (the
go:embed zero modtime emitted no validators, so the client re-downloaded
the whole bundle every launch).
- Live-stream 15s abort fix: an immediate heartbeat on open + a 10s default
interval (the first tick at 15s raced the edge idle timeout -> reconnect
storm).
PLAN/ARCHITECTURE(§13)/FUNCTIONAL(+ru)/gateway+ui+deploy READMEs updated;
round 6 closed. Tests: gateway webui/connectsrv units, ui landing unit + e2e,
full e2e (60) green.
- About screen: prominent localized title (Scrabble / Эрудит (Скрэббл)), a rules link
(en/ru Wikipedia), and the Random-game / Game-with-friends sections; copy lives in a
shared aboutContent module (the landing will reuse it). The random-game move limit
inlines the 24h auto-match clock.
- App version: Vite define __APP_VERSION__ from VITE_APP_VERSION (default 'dev'), wired as
a Docker build-arg sourced from `git describe --tags --always` in the deploy step — no
manual version bumps. The fallback keeps a plain/local build working.
- Grafana: disable Live (GF_LIVE_MAX_CONNECTIONS=0) so its WebSocket no longer trips caddy Basic-Auth and re-prompts; admin console gains a Grafana nav link
- deploy: force-recreate config-only services so reseeded Grafana dashboards / Caddyfile are actually picked up (the move-duration panel was invisible because the bind-mount went stale)
- rate-limit: raise per-user budget 120/40 -> 300/80; UI skips reloading on the echo of the player's own move (fewer requests, no double-load)
- iOS/Telegram reconnect: suppress the connection banner while backgrounded and for a short grace after resume; reconnect silently; wire visibilitychange + pageshow/pagehide + Telegram activated/deactivated (Bot API 8.0)
- hint button disabled when 0 hints remain; nudge button shows a disabled state on your own turn
- players plaque: invert so the active seat pops (accent chip, raised) and others recede
- make-move UX: a direct ✅ commit button (no hold/popover); the Shuffle tab becomes ↩️ Reset while tiles are pending
- #10 a `changes` job path-filters unit/integration/ui; an always-running `gate` job aggregates them (success-or-skipped) and becomes the only required check
- #9 deploy adds a Telegram-connector liveness probe (docker inspect: running, not restarting, stable restart count) with a VPN-handshake grace period
- #1a Game-domain dashboard gains a 'Move think-time by phase (p50/p95)' panel
- deploy README: branch protection now requires only CI / gate
Root cause of the Grafana "readdirent /etc/grafana/dashboards: no such file or
directory": the CI runner checks out into an ephemeral act workspace that is
removed after the job, so binding the compose config files straight from it
dangles the mounts in the long-lived containers (verified the act source dir is
emptied after the job). caddy/otelcol/prometheus/tempo read their config once at
startup so they survive, but would break on a restart — same latent bug.
Fix (mirrors ../galaxy-game's $HOME/.galaxy-dev/monitoring): the deploy job seeds
the config dirs to a stable $HOME/.scrabble-deploy and the compose binds them via
${SCRABBLE_CONFIG_DIR:-.} (local runs keep "."). Documented in the compose header,
deploy/README.md and the ci.yaml step.
- deploy/docker-compose.yml: mount the provisioned dashboards at
/etc/grafana/dashboards, not /var/lib/grafana/dashboards — the grafana-data
volume mounts over the latter and shadows the nested bind, so the provider
logged "readdirent /var/lib/grafana/dashboards: no such file or directory".
dashboards.yaml provider path updated to match.
- Connector telemetry stays OTLP. The VPN sidecar's netns reaches the collector's
internal IP fine (connected route, off-tunnel), but the sidecar's DNS hijacks
name resolution: AWG_CONF must NOT carry a DNS= directive, else otelcol won't
resolve ("produced zero addresses"). Without DNS= the netns uses Docker's
resolver (resolves both otelcol and api.telegram.org). Documented in
deploy/README.md (AWG_CONF row + wiring note), ARCHITECTURE §13, compose comment.
- PLAN.md: new Stage 17 "Test-contour verification & defect fixes" (exercise the
deployed contour end-to-end and fix what it surfaces — connector liveness check,
path-conditional CI); the former prod-deploy stage becomes Stage 18.
- Renumber every "Stage 17" prod-deploy reference to "Stage 18" across docs,
compose, Caddyfile, ci.yaml and CLAUDE.md; the post-Stage-14 split range is now
"Stages 15–18".
- bot.New now selects Telegram's test environment with the library's native
tgbot.UseTestEnvironment() instead of a token += "/test" hack (functionally
identical URL /bot<token>/test/METHOD, but idiomatic) + a bot test asserting
the getMe path for both test and prod.
- ci.yaml pins TELEGRAM_TEST_ENV=true for the test contour (it IS the test
environment) instead of a TEST_TELEGRAM_TEST_ENV variable: removes the
confusing double-TEST, telegram-specific, prefixed operator knob and the
secret-vs-variable footgun. Prod (Stage 17) leaves it false.
- deploy/README.md + PLAN.md updated.
- deploy/README.md documents the services, how to run it locally and in CI, and
every variable: required (the four :? ones + ≥1 bot token) and optional with
defaults, marked secret-vs-variable and with the TEST_/PROD_ Gitea mapping;
plus the fixed internal wiring and the host-side setup.
- ci.yaml maps the remaining POSTGRES_DB/USER, DICT_VERSION and LOG_LEVEL (unset
renders empty -> the compose ":-" defaults apply), so every documented var is
per-contour overridable.
- .env.example points at the README for the full reference.
- backend + gateway multi-stage distroless Dockerfiles; the gateway embeds and
serves the SPA at / and /telegram/ via go:embed (committed dist placeholder,
real build baked in by the image's node stage)
- deploy/docker-compose.yml: backend + gateway + Postgres + Telegram connector
(VPN sidecar) + OTel Collector + Prometheus (15d) + Tempo (72h) + Grafana,
fronted by a caddy owning a single /_gm Basic-Auth (admin console + Grafana
subpath); inter-service on a private network, only caddy on the edge network
- new metrics: backend accounts_created_total{kind} (robots excluded) and an
in-memory gateway active_users{window=24h,7d} gauge
- CI: single .gitea/workflows/ci.yaml (unit/integration/ui + a gated test-contour
deploy) on the new feature/* -> development -> master branch model; the old
go-unit/integration/ui-test workflows are folded in; the connector-scoped
compose is retired (superseded by deploy/)
- docs: ARCHITECTURE §11/§12/§13, root + gateway READMEs, CLAUDE.md branching,
PLAN.md (stage 16 done + refinements + Stage 17 forward-notes)
Ilia Denisov