The first v1.8.0 prod-deploy build job failed: `docker compose build` interpolates the
WHOLE compose file, and the backend's :?-guarded EXPORT_SIGN_KEY — added with the
finished-game export after the last prod release (v1.7.0), so the prod build never
exercised it — was absent from the build job's env, tripping the guard before any image
built. Prod was untouched (build-only failure: no image pushed, no deploy, no migration).
Add EXPORT_SIGN_KEY (audited every :?-guarded compose var against the build job env — it
was the only genuinely-missing one; TELEGRAM_MINIAPP_URL is derived in the run step).
Verified by reproducing the build-job env + `docker compose config`.
The maintenance overlay never showed on the test contour: only prod-deploy.sh raised
the caddy maintenance flag, so a test redeploy was a bare gateway/backend recreate — the
SPA saw a transient 502 ("Reconnecting…"), never the 503 + X-Scrabble-Maintenance marker
the overlay keys on. So the feature (PR #171) was unverifiable off prod.
Raise the flag around the recreate in ci.yaml's deploy job too, mirroring prod. Ordering
matters because of the config reseed: ci does `rm -rf $conf` + recreate, so the running
caddy sits on the stale pre-reseed bind mount and cannot see a flag written to the new
dir until it is recreated. So: raise the flag, force-recreate caddy FIRST (onto the fresh
mount, where it carries the flag and 503s), then recreate gateway/backend (the window),
then the other config services, then lower it. A trap clears the flag if the step fails,
and the next deploy's reseed wipes a stale one — so the contour can't stick in maintenance.
The caddy-routed probes run in the next step, after the flag is lowered.
Prod was already correct (prod-deploy.sh); this only makes the test contour faithful so the
overlay + reload can be seen there. An open SPA must already be on the PR-#171 bundle (which
carries the overlay code) — reload once to bootstrap onto it.
The deploy that ends a maintenance window may ship a client-incompatible change
(wire/schema bump), and the in-session bundle is the old one. On recovery, reload to
pick up the fresh client instead of just hiding the overlay. Ordering is safe: the edge
maintenance flag (deploy/prod-deploy.sh) spans the WHOLE roll and clears only at script
exit — after the gateway (which serves the embedded SPA) has rolled — so the edge 503s
everything until the entire deploy is live; recovery therefore always serves the new SPA.
A window.__maint.recover() hook + e2e cover the reload.
The edge caddy 503 carries X-Scrabble-Maintenance during a deploy window, but a user
already in the running SPA only sees their calls start failing — the static caddy page
catches only a fresh load. Detect that marker (strictly — not any transient
'unavailable', which the Connecting indicator already covers) on the raw ConnectError at
the two transport catch sites (unary exec + the live subscribe stream), before
toGatewayError discards the response headers, and raise a non-dismissable dimmed overlay
that mirrors the static caddy page. It self-clears: a capped-backoff poll (a cheap read,
mirroring connection.svelte.ts) lifts it on the first success, and a manual "retry"
button forces an immediate re-check — so it can never get stuck. On detection the read
retry loop fails fast, so a window doesn't burn the retry budget on every call.
- pure detector maintenance.ts (maintenanceRetryMs / parseRetryAfterMs) + unit tests;
the store + self-clearing poll in maintenance.svelte.ts (mirrors connection.svelte.ts)
- MaintenanceOverlay.svelte (clones Splash's fixed/inset/dimmed shell, non-dismissable),
mounted app-global in App.svelte after Coachmark
- transport.ts detects + reports at both catch sites, clears on any successful read
- i18n RU "Технические работы" / EN "Under maintenance"; a window.__maint mock hook
(the mock can't emit a real 503) + a Playwright spec
Prod is same-origin (VITE_GATEWAY_URL empty) so the marker header is readable without a
CORS expose-header. Verified: pnpm check (0), unit (402), build, e2e (186, Chromium+WebKit).
Two prod-deploy hardening measures (owner-requested):
- Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container
memory caps enforce that one service can't eat all RAM, but they overcommit the
1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel
OOM-killer (and it might pick postgres). A small swap absorbs the overshoot.
Idempotent, builtin-only (no ansible.posix).
- Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap /
migration window that the caddy edge serves a static 503 "технические работы" page
from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm
(Grafana) stays reachable. Cleared on any exit (success, health failure + rollback,
or error) by a shell trap so it can never stick on. The 503 carries Retry-After +
an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned
window apart from a transient error (the static page only catches a fresh load; an
in-session user needs the app-side overlay). Not zero-downtime — the single
stateful backend still blips — but the window is graceful instead of raw 502s.
Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC
edge and serves the page + markers; /_gm bypasses; toggling needs no reload
(per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
The planted honeytoken bearer trap already existed in the gateway + compose, but
no workflow fed GATEWAY_HONEYTOKEN, so it was always empty (inert). Wire the
per-contour TEST_/PROD_GATEWAY_HONEYTOKEN secret into the ci deploy, prod-deploy
and prod-rollback env (the prod path via the shared deploy/write-prod-env.sh),
document it (deploy/README + .env.example), and fix the stale compose comment.
Empty secret = trap off (no ":?" guard), so a deploy is safe before the operator
sets the value + plants the bait. On prod (IP ban on) presenting it earns a 24h
ban + alarm; on test (ban off) it logs + a ban metric.
GF_SMTP_ENABLED is enabled separately via the TEST_/PROD_GF_SMTP_ENABLED Gitea
variables (=true) — no code change.
Collapse identical TEST_/PROD_ pairs to single unprefixed Gitea entries, derive
the public URLs from PUBLIC_BASE_URL at deploy time, and share the prod env.sh
renderer between deploy and rollback.
- Collapse to one unprefixed variable: DICT_VERSION, SMTP_RELAY_HOST/PORT/TLS,
GRAFANA_SMTP_PORT, VITE_VK_APP_LINK, VITE_VK_APP_ID (one Selectel relay + one
pair of VK apps serve every contour). Secrets collapsed by the owner:
SMTP_RELAY_USER/PASS, GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET.
- Derive at deploy from PUBLIC_BASE_URL (no longer stored): TELEGRAM_MINIAPP_URL,
GRAFANA_ROOT_URL, VITE_VK_ID_REDIRECT_URL. Removes the prod/test asymmetry and
fills the missing test VITE_VK_APP_ID (VK web login was half-configured on test).
- Extract deploy/write-prod-env.sh + write-prod-bot-env.sh, shared by prod-deploy
and prod-rollback so the two cannot drift: a rollback now re-renders the FULL
runtime env (email / VK login / Grafana alerts previously went dark after a
rollback) and passes TELEGRAM_SUPPORT_CHAT_ID.
- Single-source DICT_VERSION (CI env + both deploys), fix the v1.3.0/v1.3.1 drift in
.env.example/README, correct the misleading honeytoken/abuse-ban compose comment,
and rewrite the deploy/README variable list (+ the previously undocumented
GATEWAY_VK_APP_SECRET and TELEGRAM_SUPPORT_CHAT_ID).
The Gitea variables are reworked via the API; the stale TEST_/PROD_ entries are
deleted after the test contour goes green.
An account merge blanket-reassigned all of the secondary's identities to the primary,
so merging two accounts that each had a confirmed email (or telegram/vk) left the
survivor with two identities of one kind — which the profile and the retention dossier
both treat as singular (the profile showed an arbitrary one; a later change-email
journaled only one). The merge now keeps the primary's identity and journals the
secondary's colliding one to retained_identities (reason=merge) before dropping it, so
the survivor has one identity per kind and the absorbed credential still lands in the
legal dossier.
- migration 00008: widen retained_identities.reason CHECK to admit 'merge'
(expand-contract — Up only widens the set, so an image rollback stays DB-safe).
- accountmerge: dedupeIdentities + retainMergedIdentity before the identity reassign.
- inttest TestAccountMergeDedupesEmail; docs ARCHITECTURE §4 + retention.go reason note.
A browser has no signed VK Mini App launch params, so linking VK on the web uses
VK ID's raw OAuth 2.1 flow (PKCE, no @vkid/sdk): the SPA redirects to VK's hosted
login and returns with an authorization code, which the gateway exchanges
server-side (confidential, under the VK "Web" app's protected key) for the trusted
vk user id — then the existing link/merge machinery attaches or merges it.
- fbs LinkVKRequest{code, device_id, code_verifier}; codec + TS bindings.
- backend link.Service ConfirmVK/MergeVK/attachVK (KindVK, mirror Telegram),
handleLinkVK[Merge], routes /user/link/vk[/merge], backendclient LinkVK[Merge].
- gateway internal/vkid confidential code exchange (id.vk.com/oauth2/auth);
transcode link.vk.confirm/merge (registered only when configured) + config
GATEWAY_VK_ID_{APP_ID,CLIENT_SECRET,REDIRECT_URL} + main wiring.
- UI lib/vkid (PKCE authorize redirect + callback), Profile "Link VK" control,
boot callback handling; a merge re-authorizes for a fresh code (VK codes are
single-use). Web-only (a redirect strands a Mini App webview).
- Deploy: VITE_VK_APP_ID + VITE_VK_ID_REDIRECT_URL build args + gateway env,
ci.yaml/prod-deploy TEST_/PROD_ vars, compose/Dockerfile/.env.example/README.
- Tests: vkid exchange unit (string/number user_id, id_token fallback, errors),
transcode link.vk, backend ConfirmVK/MergeVK inttest, codec encodeLinkVK.
- Docs: ARCHITECTURE §4, FUNCTIONAL(+ru), gateway README.
Two contour-deploy failures from PR4: (1) the deploy did not stage deploy/blackbox/, so
blackbox_exporter crash-looped on a missing config mount — add blackbox to the ci.yaml cp
and the prod-deploy tar; (2) Grafana rejects the 'Name <addr>' From form the backend go-mail
accepts and validates it even when SMTP is disabled, crash-looping Grafana — the deploy now
splits SMTP_RELAY_SERVICE_FROM into a bare GF_SMTP_FROM_ADDRESS + GRAFANA_SMTP_FROM_NAME
(handles both display and bare forms). Verified the sed split + compose render.
Drop the separate GRAFANA_SMTP_HOST (it duplicated SMTP_RELAY_HOST): Grafana differs from
the backend only in needing the STARTTLS port, so GF_SMTP_HOST is composed from
SMTP_RELAY_HOST + GRAFANA_SMTP_PORT (an explicit per-contour var, no magic default), reusing
SMTP_RELAY_USER/PASS. Wired through ci/prod/.env.example/README.
§11 gains the alerting layer: Grafana infra rules (scrape-down, edge error-rate/p99, host
mem/disk/cpu, postgres connections, TLS cert < 20d via blackbox), noDataState=OK, and the
backend admin-alert worker (coalesced email on new feedback / complaints).
Grafana: GF_SMTP from the shared relay (STARTTLS host:port) + alerting provisioning
(contact point → SERVICE_EMAIL, route-all policy, and rules for scrape-target down,
gateway internal-error rate + p99 latency, host mem/disk/cpu, postgres connections, and
TLS cert < 20 days). All rules noDataState=OK so an absent metric never false-alerts.
blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry) — effective
on prod (caddy terminates TLS; contour caddy is HTTP-only so the metric is absent).
Wires the new env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_),
prod-deploy (PROD_ + env.sh), .env.example and the README var table.
New adminalert worker polls for feedback + word complaints arriving since the last check
and coalesces a burst into one digest email per interval (5 min), inert unless a distinct
admin sender (BACKEND_SMTP_ADMIN_FROM) and recipient (BACKEND_ADMIN_EMAIL, comma-separated
allowed) are configured. The mailer gains a per-message From override and splits a
comma-separated To into separate recipients (go-mail needs them as a list). Feedback/game
stores gain CountSince/CountComplaintsSince. Unit tests cover the digest, the skip-when-
empty, and the recipient split.
The email/name/external-id search was scoped to one tab and only looked in the live
identities table, so a deleted account (whose credentials moved to retained_identities on
deletion) could not be found by the email/id it held. Now a people search spans live and
deleted accounts in one query (robots only on the Robots tab) and also matches the
retention journal and the retained real name; results carry a 'deleted' badge. Integration
test: a deleted account is found by its held email and external id.
- Admin /users gains a Deleted scope (between People and Robots); every other scope now
hides tombstoned accounts (deleted_at filter in UserFilter).
- Admin delete-user is offered for any non-deleted account (drop the not-guest gate, so a
stale is_guest account can still be tombstoned).
- Harden EmailService.ConfirmCode to ClearGuest — defence-in-depth so no confirmed-email
path leaves is_guest set (the currently-live paths already do).
- Space the delete/change dialog's action row from its input field.
Integration tests: the Deleted filter scoping + ConfirmCode guest promotion.
FUNCTIONAL (+ru): the deletion user story — anonymised live surfaces ([Deleted]),
freed sign-in methods, a two-year-retained dossier, the code/phrase step-up, game
forfeit + all-robot drop, fresh account on reopen. ARCHITECTURE: the retention model
(retained_identities journal on every detach, tombstone + [Deleted] sentinel, drop
all-robot games, purpose=delete step-up, cold-load last-login, two-year TTL reaper).
The user-detail console gains a Deletion & retention panel: last login (time + IP),
the tombstone (deleted-at + retained real name), and the retention journal (the legal
dossier of detached credentials). A Delete-user action runs the same deletion
orchestration as the in-app flow (mirrors the email-erase pattern). Store readers
RetainedIdentities + DeletionInfo back the view; integration test covers them.
Profile gains a Delete-account control (durable accounts) opening a step-up dialog: a
mailed code for an email account, or the typed DELETE phrase for a platform-only one.
On success the app swaps to a terminal AccountDeleted screen ('Учётная запись удалена')
with a Close that closes the host Mini App (telegramClose / vkClose; web = no close).
Wires deleteRequest/deleteConfirm through client/transport/mock/codec; ru/en i18n;
codec wire test + Chromium/WebKit e2e.
Step-up: email accounts confirm with a mailed code (purpose=delete, no deeplink —
ConfirmByToken refuses a delete token so a stray click can't delete); platform-only
accounts type a fixed phrase (anti-impulse). Endpoints /user/delete/{request,confirm};
the confirm orchestration resigns active games, drops all-robot games, tombstones +
anonymizes the account (freeing its creds), and revokes its sessions — the tombstone is
the point of no return, the rest best-effort. Gateway account.delete.{request,confirm}
ops + fbs AccountDeleteConfirm/AccountDeleteRequestResult + branded ru/en delete email.
Integration tests cover the step-up (code + no-email) and the orchestration pieces.
Q2=B: DropAllRobotGames deletes the deletee's games with no human opponent (vs-AI or
auto-match-robot; children cascade), keeping games with any human seat (anonymized
instead). Q1=A: the anon label is the sentinel [Deleted] — the editable-name rule forbids
brackets, so no live player can impersonate a deleted account. Integration test covers
drop-vs-keep.
Daily background reaper purges the deletion dossier past its two-year TTL: every
retained_identities row by detached_at (covering unlink/change on live accounts too),
and — for accounts tombstoned before the cutoff — the retained feedback thread plus the
dossier PII (deleted_display_name, last_login_ip). Chat is kept (a shared game artifact)
and the tombstone row stays. Started from main next to the guest reaper. Integration
test covers the cutoff boundary and the deleted-account feedback/PII purge.
New accountdelete package: AnonymizeAndTombstone journals every credential to the
retention log (reason=delete) then removes them (freeing email/vk/tg for reuse),
snapshots the real display name into deleted_display_name and scrubs the live one to
'Удалённый игрок', sets deleted_at, anonymizes the account's game-seat snapshots, and
drops its friendships/blocks/invitations/friend-codes/drafts/pending-codes — all in one
transaction. Chat, feedback and complaints are kept (the tombstone keeps their
no-cascade FKs valid). Session revocation + game forfeit are orchestrated a layer up.
Integration test covers journalling, tombstone/scrub and credential reuse.
The profile GET (fetched once per cold app-load by the SPA) stamps accounts
.last_login_at/.last_login_ip, throttled to at most once per hour per account
(best-effort, never blocks the read). IP from the gateway-forwarded X-Forwarded-For.
Feeds the account-deletion dossier. Integration test covers the throttle.
On unlink (RemoveIdentity, reason=unlink) and email change (replaceEmailIdentity,
reason=change) write the outgoing credential to retained_identities before removing
the live identities row — so the legal dossier survives while the (kind, external_id)
frees for reuse. Same transaction, so the dossier and live state cannot diverge.
Integration tests cover both reasons.
Additive/expand-contract: new append-only retained_identities table (the legal
dossier of every detached credential, keyed by account_id, TTL'd by detached_at)
plus nullable accounts columns last_login_at/last_login_ip (cold-load stamp) and
deleted_at/deleted_display_name (tombstone + retained real name). Regenerates the
go-jet models for accounts + retained_identities only.
Replace the stale skipped linking specs: change-email refuses a taken address with
the non-disclosing message (never revealing the other account) and replaces a free
one; the web Telegram link control links then unlinks through the confirm dialog.
Chromium + WebKit.
FUNCTIONAL (+ru): drop the stale 'linking UI hidden' note; document the profile
sign-in-methods matrix, the unlink last-identity guard, and the non-disclosing
atomic email change. ARCHITECTURE: unlink + change-email behaviour and the
purpose=change deeplink branch.
Integration: unlinking a provider keeps the other identities and refuses removing
the last one; change-email replaces the address (freeing the old), refuses a taken
address without merging, and works through the one-tap deeplink token. Unit: the
change-email template renders localised ru/en copy.
Profile shows the account's sign-in methods for guests and durable accounts alike:
add or change email, link Telegram on the web (login widget), and unlink a linked
provider. Email is never unlinked (it is changed); unlink is offered only when
another identity remains, and a change to an address owned by another account shows
the non-disclosing 'check the address or contact support'. Add-VK-on-web stays
deferred (no VK OAuth).
Wires linkUnlink / changeEmailRequest / changeEmailConfirm through the client,
transport, mock and codec (encodeLinkUnlink + LinkResult 'unlinked'/'changed'
statuses); codec wire tests; ru/en i18n.
Unlink: POST /user/link/unlink (telegram|vk) via account.RemoveIdentity, refusing
the last identity; email is never unlinked. New fbs LinkUnlinkRequest + gateway
link.unlink op, returning the refreshed profile.
Change-email: purposeChange confirm-codes (RequestChangeCode/ConfirmChange) that
atomically replace the account's confirmed email (account.replaceEmailIdentity);
a new address owned by another account is refused without disclosure, never merged.
The one-tap deeplink handles purposeChange too. Reuses the LinkEmail* fbs tables;
gateway link.email.change.{request,confirm} ops + backendclient methods; branded
ru/en change-email copy.
Generalize the email-erase store op to any identity kind (with the same
last-identity guard and, for email, the pending-confirmation cleanup) so the
profile Unlink control can reuse it for Telegram/VK. RemoveEmailIdentity stays as a
thin wrapper for the admin console.
Add email / telegram_linked / vk_linked to the Profile (fbs table + regenerated
Go/TS bindings, gateway ProfileResp + encodeProfile, backend DTO, UI model +
decode). They are filled outside the pure projection — Server.profileResponse now
reads the account's identities (like the banner seam) — and will drive the profile's
Add / Unlink / change-email controls.
The session-less /confirm page defaulted to English, so it showed the English
app title. Carry the recipient's language on the deeplink (?lang) and setLocale on
load so the page matches the email. Show a localised brand wordmark (Эрудит / Erudit,
matching the email) on the success state only; the invalid/expired state now shows
just the message, no header.
Add an exact (strict) email filter to the /users list (UserFilter.EmailExact →
a kind='email' identity match) with a search input, and an 'Erase email' action on
the user card that deletes the bound email identity and its pending confirmations,
freeing the address. It refuses to remove the account's only identity
(ErrLastIdentity), which would leave it unreachable. Integration tests for both.
Drop the confirm button: the /confirm screen confirms the token as soon as it
loads. The token rides the URL fragment (never sent to the server), so a plain link
prefetch cannot reach it, and the manual six-digit code is the fallback if an
aggressive scanner runs the page. Update the ARCHITECTURE/FUNCTIONAL wording
accordingly and swap the confirm.prompt/action strings for confirm.busy (en+ru).
ConfirmByToken attached the confirmed email to the account but, on the link path,
skipped ClearGuest — so a guest who bound an email via the one-tap deeplink stayed a
guest (the code-based flow clears it in the link service). Clear the flag on a free
link too, promoting the guest to a durable account; the profile live event then
refreshes the open session. Integration test added.
Ilia Denisov
developer