Merge pull request 'R2: stress harness + contour resource observability + early run' (#33) from feature/r2-loadtest-observability into development
CI / changes (push) Successful in 1s
CI / integration (push) Successful in 11s
CI / unit (push) Successful in 9s
CI / ui (push) Successful in 36s
CI / gate (push) Successful in 0s
CI / deploy (push) Successful in 55s

This commit was merged in pull request #33.
This commit is contained in:
2026-06-09 23:01:30 +00:00
32 changed files with 2766 additions and 15 deletions
+4 -4
View File
@@ -64,7 +64,7 @@ jobs:
if [ "$files" != "__DIFF_FAILED__" ]; then
echo "changed files:"; echo "$files"
go=false; ui=false
if echo "$files" | grep -qE '^(backend/|pkg/|gateway/|platform/|go\.work)'; then go=true; fi
if echo "$files" | grep -qE '^(backend/|pkg/|gateway/|platform/|loadtest/|go\.work)'; then go=true; fi
if echo "$files" | grep -qE '^ui/'; then ui=true; fi
# A workflow or deploy change re-runs everything as a safety net.
if echo "$files" | grep -qE '^(\.gitea/workflows/|deploy/)'; then go=true; ui=true; fi
@@ -112,15 +112,15 @@ jobs:
fi
- name: vet
run: go vet ./backend/... ./pkg/... ./gateway/... ./platform/telegram/...
run: go vet ./backend/... ./pkg/... ./gateway/... ./platform/telegram/... ./loadtest/...
- name: build
run: go build ./backend/... ./pkg/... ./gateway/... ./platform/telegram/...
run: go build ./backend/... ./pkg/... ./gateway/... ./platform/telegram/... ./loadtest/...
- name: test
env:
BACKEND_DICT_DIR: ${{ github.workspace }}/dawg
run: go test -count=1 ./backend/... ./pkg/... ./gateway/... ./platform/telegram/...
run: go test -count=1 ./backend/... ./pkg/... ./gateway/... ./platform/telegram/... ./loadtest/...
integration:
needs: changes
+3 -2
View File
@@ -126,8 +126,9 @@ backend/ # module scrabble/backend
docs/ .gitea/workflows/ PLAN.md CLAUDE.md README.md
gateway/ ui/ pkg/ # added by their stages
platform/telegram/ # Telegram connector side-service (Stage 9): bot + gRPC API
backend/Dockerfile gateway/Dockerfile platform/telegram/Dockerfile # multi-stage distroless (Stage 16)
deploy/ # docker-compose + caddy + otelcol/prometheus/tempo/grafana (Stage 16)
loadtest/ # module scrabble/loadtest: the pre-release stress harness (R2)
backend/Dockerfile gateway/Dockerfile platform/telegram/Dockerfile loadtest/Dockerfile # multi-stage distroless (Stage 16; loadtest R2)
deploy/ # docker-compose + caddy + otelcol/prometheus/tempo/grafana (+ cAdvisor/postgres_exporter, R2)
```
## Build & test
+34 -1
View File
@@ -18,7 +18,7 @@ the edge before prod. Each phase maps back to the owner's raw pre-release TODO l
| # | Phase | Raw TODOs | Status |
|---|-------|-----------|--------|
| R1 | Schema & naming reset | 1 + 10 | **done** |
| R2 | Stress harness + contour observability + early run | 9a | todo |
| R2 | Stress harness + contour observability + early run | 9a | **done** |
| R3 | Edge hardening | 2 + 8 + 3 | todo |
| R4 | Push enrichment + kill the last poll | 4 + 5 | todo |
| R5 | Bundle slimming | 6 | todo |
@@ -220,3 +220,36 @@ Then Stage 18.
contour `backend` schema was wiped (`DROP SCHEMA backend CASCADE` + restart, not a volume drop) and
re-migrated to the baseline — verified the new variant CHECK (`scrabble_en/scrabble_ru/erudit_ru`),
`games`=0 and a clean boot.
- **R2** (interview + implementation):
- **Locked decisions:** game assembly via **invitations** (real path, no robots; not direct game-row
inserts); **moderate** ramp **50 → 200 → 500** at 10 min/step; **diagnostic** pass bar (no SLO gate);
run as a **one-shot container on `scrabble-internal`** in this PR.
- **Harness** = new `scrabble/loadtest` module (`use ./loadtest` + a `replace scrabble/gateway` for the
dot-free edge-proto import). It seeds 1000 guest + 10000 durable accounts + sessions **directly in
Postgres** (token hash mirrors `backend/internal/session`), drives players over the **edge protocol**,
generates **mid-ranked legal moves locally** with the embedded `scrabble-solver` by replaying
`game.history` (the edge carries no board — mirrors `engine.ReplayBoard` via the public API), and a
**gateway-hammer**. Compact CLI (`run` / `cleanup`), distroless Dockerfile (DAWGs baked), Go unit tests.
- **Adding the module broke the other images' builds** — backend/gateway/telegram Dockerfiles reduce the
workspace but still referenced `./loadtest` (not in their context); each now also
`-dropuse=./loadtest` (backend/telegram additionally `-dropreplace` the gateway replace). Caught by the
first deploy run; verified by building all four images.
- **Harness payload fixes found by the smoke pass:** the draft DTO's `rack_order` is a string (was sent
as `[]``bad_request`); the display-name validator forbids digits/colons, so the cleanup marker
became a letters-only `Zzloadtest` so `profile.update` resends the seeded name. `chat_not_your_turn` /
`nudge_own_turn` are **by-design** turn gates, correctly exercised.
- **Observability:** added **cAdvisor + postgres_exporter** + the **Scrabble — Resources** dashboard +
two Prometheus jobs. **Finding:** cAdvisor yields only the root cgroup on the contour host (separate
XFS `/var/lib/docker` breaks its layer-ID resolution — the existing galaxy deploy has the same limit),
so per-container CPU/RSS for the early pass was captured via `docker stats`. **R7:** adopt the otelcol
`docker_stats` receiver (already the contrib image) for per-container metrics in Grafana.
- **Early run (2026-06-09):** ramped clean to 500 players, no crash/deadlock, cleanup removed all 11000
accounts. 1.2 M edge calls, 48 870 plays, 2 798 games finished; the per-user limiter held under the
hammer (99.97 % rejected, p99 2 ms). **Top finding:** ~14 % `transport_error` on `game.state` at 500
players, under CPU saturation (backend/gateway/Postgres each ~1 core) and amplified by the harness's
single shared `http2.Transport`; the harness itself peaked at 86 % of a core on the same host, so the
figures are pessimistic. Full trip report in [`../loadtest/REPORT-R2.md`](../loadtest/REPORT-R2.md);
it feeds R3 (h2c `MaxConcurrentStreams`/timeouts, body-size cap), R6 and R7 (per-player transports,
separate hardware, pool/limit sizing).
- **CI:** `./loadtest/...` added to the path filter + vet/build/test; `go.work.sum` carries the new deps.
+3 -2
View File
@@ -30,8 +30,9 @@ COPY go.work go.work.sum ./
COPY pkg ./pkg
COPY backend ./backend
# Reduce the workspace to what the backend needs: backend + pkg.
RUN go work edit -dropuse=./gateway -dropuse=./platform/telegram
# Reduce the workspace to what the backend needs: backend + pkg. loadtest and the
# gateway replace it requires are not in this context, so drop both.
RUN go work edit -dropuse=./gateway -dropuse=./platform/telegram -dropuse=./loadtest -dropreplace=scrabble/gateway@v0.0.0
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/backend ./backend/cmd/backend
# --- runtime -----------------------------------------------------------------
+32
View File
@@ -225,6 +225,38 @@ services:
- grafana-data:/var/lib/grafana
networks: [internal]
# cAdvisor exports per-container resource metrics (CPU / memory / network / disk)
# for the R2/R7 stress runs' resource baseline. Prometheus scrapes it at :8080
# over the internal network. It needs read access to the host's cgroup and
# container state; --docker_only trims non-container cgroup series.
cadvisor:
container_name: scrabble-cadvisor
image: gcr.io/cadvisor/cadvisor:v0.49.1
restart: unless-stopped
privileged: true
command: ["--docker_only=true", "--housekeeping_interval=15s"]
devices:
- /dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
networks: [internal]
# postgres_exporter exports Postgres server metrics (connections, cache hit ratio,
# transactions, database size). Prometheus scrapes it at :9187. The DSN reuses the
# contour Postgres credentials; sslmode=disable on the internal network.
postgres_exporter:
container_name: scrabble-postgres-exporter
image: prometheuscommunity/postgres-exporter:v0.16.0
restart: unless-stopped
depends_on: [postgres]
environment:
DATA_SOURCE_NAME: postgresql://${POSTGRES_USER:-scrabble}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-scrabble}?sslmode=disable
networks: [internal]
networks:
internal:
name: scrabble-internal
+84
View File
@@ -0,0 +1,84 @@
{
"uid": "scrabble-resources",
"title": "Scrabble — Resources",
"tags": ["scrabble"],
"timezone": "",
"schemaVersion": 39,
"version": 1,
"refresh": "30s",
"time": { "from": "now-1h", "to": "now" },
"panels": [
{
"type": "stat",
"title": "Postgres connections",
"description": "Backends connected to the scrabble database (postgres_exporter).",
"gridPos": { "h": 5, "w": 6, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [{ "refId": "A", "expr": "sum(pg_stat_database_numbackends{datname=\"scrabble\"})" }]
},
{
"type": "stat",
"title": "Postgres cache hit ratio",
"description": "blks_hit / (blks_hit + blks_read) over 5m.",
"gridPos": { "h": 5, "w": 6, "x": 6, "y": 0 },
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [{ "refId": "A", "expr": "sum(rate(pg_stat_database_blks_hit{datname=\"scrabble\"}[5m])) / clamp_min(sum(rate(pg_stat_database_blks_hit{datname=\"scrabble\"}[5m])) + sum(rate(pg_stat_database_blks_read{datname=\"scrabble\"}[5m])), 1)" }]
},
{
"type": "stat",
"title": "Postgres commits/s",
"gridPos": { "h": 5, "w": 6, "x": 12, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [{ "refId": "A", "expr": "sum(rate(pg_stat_database_xact_commit{datname=\"scrabble\"}[5m]))" }]
},
{
"type": "stat",
"title": "Database size",
"gridPos": { "h": 5, "w": 6, "x": 18, "y": 0 },
"fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [{ "refId": "A", "expr": "max(pg_database_size_bytes{datname=\"scrabble\"})" }]
},
{
"type": "timeseries",
"title": "Container CPU (cores) by container",
"description": "cAdvisor container_cpu_usage_seconds_total rate, per scrabble-* container (the load harness appears when run as --name scrabble-loadtest). Verify the metric name against live Prometheus if empty.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 5 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [{ "refId": "A", "expr": "sum(rate(container_cpu_usage_seconds_total{name=~\"scrabble-.+\"}[5m])) by (name)", "legendFormat": "{{name}}" }]
},
{
"type": "timeseries",
"title": "Container memory (working set) by container",
"description": "cAdvisor container_memory_working_set_bytes, per scrabble-* container.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 5 },
"fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [{ "refId": "A", "expr": "max(container_memory_working_set_bytes{name=~\"scrabble-.+\"}) by (name)", "legendFormat": "{{name}}" }]
},
{
"type": "timeseries",
"title": "Container network I/O by container",
"description": "cAdvisor receive (+) and transmit (-) byte rates per scrabble-* container.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
"fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{ "refId": "A", "expr": "sum(rate(container_network_receive_bytes_total{name=~\"scrabble-.+\"}[5m])) by (name)", "legendFormat": "rx {{name}}" },
{ "refId": "B", "expr": "-sum(rate(container_network_transmit_bytes_total{name=~\"scrabble-.+\"}[5m])) by (name)", "legendFormat": "tx {{name}}" }
]
},
{
"type": "timeseries",
"title": "Postgres transactions/s",
"description": "Commit and rollback rates on the scrabble database (postgres_exporter).",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{ "refId": "A", "expr": "sum(rate(pg_stat_database_xact_commit{datname=\"scrabble\"}[5m]))", "legendFormat": "commit" },
{ "refId": "B", "expr": "sum(rate(pg_stat_database_xact_rollback{datname=\"scrabble\"}[5m]))", "legendFormat": "rollback" }
]
}
]
}
+9
View File
@@ -12,3 +12,12 @@ scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["localhost:9090"]
# Container resource metrics (CPU/memory/network/disk) for every contour
# container, for the R2/R7 stress runs' resource baseline.
- job_name: cadvisor
static_configs:
- targets: ["cadvisor:8080"]
# Postgres server metrics (connections, cache hit ratio, transactions, db size).
- job_name: postgres_exporter
static_configs:
- targets: ["postgres_exporter:9187"]
+5 -1
View File
@@ -541,7 +541,11 @@ promotions) is future work and would deliver short markdown messages (text + lin
metrics + Tempo traces), **Prometheus** (15d), **Tempo** (72h) and **Grafana**
(provisioned datasources + dashboards, behind the caddy `/_gm/grafana` Basic-Auth)
are stood up with the deploy (`deploy/`, Stage 16); the default exporter stays
`none`, so CI needs no collector.
`none`, so CI needs no collector. The contour also runs **cAdvisor** (per-container
CPU/memory/network) and **postgres_exporter** (connections, cache-hit ratio,
transactions, db size), scraped by Prometheus and surfaced on the **Scrabble —
Resources** Grafana dashboard (R2), so the pre-release stress runs capture a resource
baseline; these export directly in Prometheus format (not through the collector).
- Per-request server-side timing via gin middleware from day one (the access log
carries method, route, status, latency and the active trace id). A
client-measured RTT piggybacked on the next request is a later enhancement.
+14
View File
@@ -103,6 +103,20 @@ tests or touching CI.
`otlp` now accepted, an unsupported exporter rejected) and the guest-reaper knobs.
Postgres-backed `inttest` drives the **guest reaper** end to end (an abandoned guest is
reaped; a too-young guest, a seated guest and a durable account are kept).
- **Load test & resource baseline** *(R2)* — a reusable `loadtest/` module
(`scrabble/loadtest`) is the pre-release stress harness. It **seeds** a large account
population with pre-created sessions directly in Postgres (token hashes matching
`backend/internal/session`), **drives** virtual players through the edge protocol —
real games assembled via invitations, **mid-ranked** legal moves generated locally by
the embedded `scrabble-solver` (the edge carries no board, so the client replays
history) — plus a fraction of nudge/chat/check-word/draft/profile/stats ops, and a
**gateway-hammer** that verifies the rate limiter. Its own Go unit tests cover the pure
pieces (token hashing, board replay vs. `board.Parse`, rack reconstruction, mid-rank
selection, the report); the DAWG-backed move test runs under `BACKEND_DICT_DIR` (as the
engine tests do). It is **not** part of the per-PR suite's behavioural assertions: it
runs ad hoc as a one-shot container against the contour, producing a trip report (bugs
+ a resource baseline) read off the **cAdvisor + postgres_exporter** Grafana dashboard
added to the contour in R2. See [`../loadtest/README.md`](../loadtest/README.md).
## Principles
+3 -2
View File
@@ -50,8 +50,9 @@ COPY gateway ./gateway
RUN rm -rf gateway/internal/webui/dist
COPY --from=ui /ui/dist gateway/internal/webui/dist
# Reduce the workspace to what the gateway needs: gateway + pkg.
RUN go work edit -dropuse=./backend -dropuse=./platform/telegram
# Reduce the workspace to what the gateway needs: gateway + pkg (loadtest is not in
# this context; its scrabble/gateway replace targets ./gateway, which is present here).
RUN go work edit -dropuse=./backend -dropuse=./platform/telegram -dropuse=./loadtest
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/gateway ./gateway/cmd/gateway
# --- runtime -----------------------------------------------------------------
+6
View File
@@ -4,6 +4,7 @@ use ./backend
use (
./gateway
./loadtest
./pkg
./platform/telegram
)
@@ -19,3 +20,8 @@ use (
// like scrabble-solver it cannot be fetched as a versioned dependency; the
// replace points the v0.0.0 require at the in-repo module directory.
replace scrabble/pkg v0.0.0 => ./pkg
// scrabble/gateway is required by the loadtest harness for the generated edge
// Connect client and proto envelope (gateway/proto/edge). Same dot-free reason as
// scrabble/pkg — the replace points its v0.0.0 require at the in-repo directory.
replace scrabble/gateway v0.0.0 => ./gateway
+99
View File
@@ -1,38 +1,78 @@
cel.dev/expr v0.25.1 h1:1KrZg61W6TWSxuNZ37Xy49ps13NUovb66QLprthtwi4=
cel.dev/expr v0.25.1/go.mod h1:hrXvqGP6G6gyx8UAHSHJ5RGk//1Oj5nXQ2NI02Nrsg4=
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
gitea.iliadenisov.ru/developer/scrabble-solver v1.0.0 h1:ntN6m4cOB+4FelleO2nkAIZp8WSc+v25neetzfdUuuw=
gitea.iliadenisov.ru/developer/scrabble-solver v1.0.0/go.mod h1:G60OiGZtkrRyYX8P3SSsjVpU707fufmZkvCkNFPFWrY=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/ClickHouse/ch-go v0.71.0 h1:bUdZ/EZj/LcVHsMqaRUP2holqygrPWQKeMjc6nZoyRM=
github.com/ClickHouse/ch-go v0.71.0/go.mod h1:NwbNc+7jaqfY58dmdDUbG4Jl22vThgx1cYjBw0vtgXw=
github.com/ClickHouse/clickhouse-go/v2 v2.45.0 h1:iHt15nA4iYhfde5bDQAcLAat9BAh7B5ksPRNRa4UI7s=
github.com/ClickHouse/clickhouse-go/v2 v2.45.0/go.mod h1:giJfUVlMkcfUEPVfRpt51zZaGEx9i17gCos8gBl392c=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0 h1:DHa2U07rk8syqvCge0QIGMCE1WxGj9njT44GH7zNJLQ=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0/go.mod h1:P4WPRUkOhJC13W//jWpyfJNDAIpvRbAUIYLX/4jtlE0=
github.com/Masterminds/semver/v3 v3.1.1 h1:hLg3sBzpNErnxhQtUy/mmLR2I9foDujNK030IGemrRc=
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
github.com/andybalholm/brotli v1.2.1/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/antihax/optional v1.0.0 h1:xK2lYat7ZLaVVcIuj82J8kIro4V6kDe0AUDFboUCwcg=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ=
github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw=
github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5 h1:6xNmx7iTtyBRev0+D/Tv1FZd4SCg8axKApyNyRsAt/w=
github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5/go.mod h1:KdCmV+x/BuvyMxRnYBlmVaq4OLiKW6iRQfvC62cvdkI=
github.com/cockroachdb/apd v1.1.0 h1:3LFP3629v+1aKXU5Q37mxmRxX/pIu1nijXydLShEq5I=
github.com/coder/websocket v1.8.14 h1:9L0p0iKiNOibykf283eHkKUHHrpG7f65OE3BhhO7v9g=
github.com/coder/websocket v1.8.14/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
github.com/containerd/typeurl/v2 v2.2.0 h1:6NBDbQzr7I5LHgp34xAXYF5DOTQDn05X58lsPEmzLso=
github.com/containerd/typeurl/v2 v2.2.0/go.mod h1:8XOOxnyatxSWuG8OfsZXVnAF4iZfedjS/8UHSPJnX4g=
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f h1:JOrtw2xFKzlg+cbHpyrpLDmnN1HqhBfnX7WDiW7eG2c=
github.com/elastic/go-sysinfo v1.15.4 h1:A3zQcunCxik14MgXu39cXFXcIw2sFXZ0zL886eyiv1Q=
github.com/elastic/go-sysinfo v1.15.4/go.mod h1:ZBVXmqS368dOn/jvijV/zHLfakWTYHBZPk3G244lHrU=
github.com/elastic/go-windows v1.0.2 h1:yoLLsAsV5cfg9FLhZ9EXZ2n2sQFKeDYrHenkcivY4vI=
github.com/elastic/go-windows v1.0.2/go.mod h1:bGcDpBzXgYSqM0Gx3DM4+UxFj300SZLixie9u9ixLM8=
github.com/envoyproxy/go-control-plane v0.14.0 h1:hbG2kr4RuFj222B6+7T83thSPqLjwBIfQawTkC++2HA=
github.com/envoyproxy/go-control-plane v0.14.0/go.mod h1:NcS5X47pLl/hfqxU70yPwL9ZMkUlwlKxtAohpi2wBEU=
github.com/envoyproxy/go-control-plane/envoy v1.36.0 h1:yg/JjO5E7ubRyKX3m07GF3reDNEnfOboJ0QySbH736g=
github.com/envoyproxy/go-control-plane/envoy v1.36.0/go.mod h1:ty89S1YCCVruQAm9OtKeEkQLTb+Lkz0k8v9W0Oxsv98=
github.com/envoyproxy/go-control-plane/ratelimit v0.1.0 h1:/G9QYbddjL25KvtKTv3an9lx6VBE2cnb8wp1vEGNYGI=
github.com/envoyproxy/go-control-plane/ratelimit v0.1.0/go.mod h1:Wk+tMFAFbCXaJPzVVHnPgRKdUdwW/KdbRt94AzgRee4=
github.com/envoyproxy/protoc-gen-validate v1.3.0 h1:TvGH1wof4H33rezVKWSpqKz5NXWg5VPuZ0uONDT6eb4=
github.com/envoyproxy/protoc-gen-validate v1.3.0/go.mod h1:HvYl7zwPa5mffgyeTUHA9zHIH36nmrm7oCbo4YKoSWA=
github.com/felixge/fgprof v0.9.3 h1:VvyZxILNuCiUCSXtPtYmmtGvb65nqXh2QFWc0Wpf2/g=
github.com/felixge/fgprof v0.9.3/go.mod h1:RdbpDgzqYVh/T9fPELJyV7EYJuHB55UTEULNun8eiPw=
github.com/friendsofgo/errors v0.9.2 h1:X6NYxef4efCBdwI7BgS820zFaN7Cphrmb+Pljdzjtgk=
github.com/friendsofgo/errors v0.9.2/go.mod h1:yCvFW5AkDIL9qn7suHVLiI/gH228n7PC4Pn44IGoTOI=
github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw=
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
github.com/go-jose/go-jose/v4 v4.1.3 h1:CVLmWDhDVRa6Mi/IgCgaopNosCaHz7zrMeF9MlZRkrs=
github.com/go-jose/go-jose/v4 v4.1.3/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-kit/log v0.1.0 h1:DGJh0Sm43HbOeYDNnVZFl8BvcYVvjD5bqYJvp0REbwQ=
github.com/go-logfmt/logfmt v0.5.0 h1:TrB8swr/68K7m9CcGut2g3UOihhbcbiMAYiuTXdEih4=
github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo=
github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU=
github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk=
github.com/go-telegram/bot v1.21.0 h1:Va/PbGc2vBDdv57GCUEEVV6ROlHWiC6SklJY9Hvhzps=
github.com/go-telegram/bot v1.21.0/go.mod h1:i2TRs7fXWIeaceF3z7KzsMt/he0TwkVC680mvdTFYeM=
github.com/gofrs/uuid v4.0.0+incompatible h1:1SD/1F5pU8p29ybwgQSwpQk+mwdRrXCYuPhW6m+TnJw=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI=
github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 h1:au07oEsX2xN0ktxqI+Sida1w446QrXBRJ0nee3SNZlA=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
github.com/golang-sql/sqlexp v0.1.0 h1:ZCD6MBpcuOVfGVqsEmY5/4FtYiKz6tSyUv9LPEDei6A=
github.com/golang-sql/sqlexp v0.1.0/go.mod h1:J4ad9Vo8ZCWQ2GMrC4UCQy1JpCbwU9m3EOqtpKwwwHI=
github.com/golang/glog v1.2.5 h1:DrW6hGnjIhtvhOIiAKT6Psh/Kd/ldepEa81DKeiRJ5I=
github.com/golang/glog v1.2.5/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/google/gofuzz v1.0.0 h1:A8PeW59pxE9IoFRqBp37U+mSNaQoZ46F1f0f863XSXw=
github.com/google/pprof v0.0.0-20211214055906-6f57359322fd h1:1FjCyPC+syAzJ5/2S8fqdZK1R22vvA0J7JZKcuOIQ7Y=
github.com/google/pprof v0.0.0-20211214055906-6f57359322fd/go.mod h1:KgnwoLYCZ8IQu3XUZ8Nc/bM9CCZFOyjUNOSygVozoDg=
github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA=
github.com/iliadenisov/alphabet v1.1.0 h1:d87N7Rmpjj9FgL7bvEaqLdaIaNch2hC6HvkbKGhn7Hk=
github.com/iliadenisov/alphabet v1.1.0/go.mod h1:h6BhDBiJBLhMEb5XfsqJXZop3hhwXaD8lc5yf38Baqw=
github.com/iliadenisov/dafsa v1.1.0 h1:NV1ZOstMdHXI/cCyAZKOD3qnKLoYdMUunA0+Baj7vR4=
@@ -40,59 +80,118 @@ github.com/iliadenisov/dafsa v1.1.0/go.mod h1:mG6Y0DdfRrqdXGqTEMb9Zx0Fl0NkP3ZDYe
github.com/jackc/chunkreader v1.0.0 h1:4s39bBR8ByfqH+DKm8rQA3E1LHZWB9XWcrz8fqaZbe0=
github.com/jackc/pgproto3 v1.1.0 h1:FYYE4yRw+AgI8wXIinMlNjBbp/UitDJwfj5LqqewP1A=
github.com/jackc/puddle v1.3.0 h1:eHK/5clGOatcjX3oWGBO/MpxpbHzSwud5EWTSCI+MX0=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/jonboulle/clockwork v0.5.0 h1:Hyh9A8u51kptdkR+cqRpT1EebBwTn1oK9YfGYbdFz6I=
github.com/jonboulle/clockwork v0.5.0/go.mod h1:3mZlmanh0g2NDKO5TWZVJAfofYk64M7XN3SzBPjZF60=
github.com/jordanlewis/gcassert v0.0.0-20250430164644-389ef753e22e h1:a+PGEeXb+exwBS3NboqXHyxarD9kaboBbrSp+7GuBuc=
github.com/jordanlewis/gcassert v0.0.0-20250430164644-389ef753e22e/go.mod h1:ZybsQk6DWyN5t7An1MuPm1gtSZ1xDaTXS9ZjIOxvQrk=
github.com/kisielk/gotool v1.0.0 h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg=
github.com/klauspost/compress v1.17.6/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
github.com/konsorten/go-windows-terminal-sequences v1.0.2 h1:DB17ag19krx9CFsz4o3enTrPXyIXCl+2iCXH/aMAp9s=
github.com/kr/pty v1.1.8 h1:AkaSdXYQOWeaO3neb8EM634ahkXXe3jYbVh/F9lq+GI=
github.com/mattn/go-colorable v0.1.6 h1:6Su7aK7lXmJ/U79bYtBjLNaha4Fs1Rg9plHpcH+vvnE=
github.com/mattn/go-sqlite3 v1.14.28 h1:ThEiQrnbtumT+QMknw63Befp/ce/nUPgBPMlRFEum7A=
github.com/mattn/go-sqlite3 v1.14.28/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/mfridman/xflag v0.1.0 h1:TWZrZwG1QklFX5S4j1vxfF1sZbZeZSGofMwPMLAF29M=
github.com/mfridman/xflag v0.1.0/go.mod h1:/483ywM5ZO5SuMVjrIGquYNE5CzLrj5Ux/LxWWnjRaE=
github.com/microsoft/go-mssqldb v1.9.8 h1:d4IFMvF/o+HdpXUqbBfzHvn/NlFA75YGcfHUUvDFJEM=
github.com/microsoft/go-mssqldb v1.9.8/go.mod h1:eGSRSGAW4hKMy5YcAenhCDjIRm2rhqIdmmwgciMzLus=
github.com/moby/sys/mount v0.3.4 h1:yn5jq4STPztkkzSKpZkLcmjue+bZJ0u2AuQY1iNI1Ww=
github.com/moby/sys/mount v0.3.4/go.mod h1:KcQJMbQdJHPlq5lcYT+/CjatWM4PuxKe+XLSVS4J6Os=
github.com/moby/sys/mountinfo v0.7.2 h1:1shs6aH5s4o5H2zQLn796ADW1wMrIwHsyJ2v9KouLrg=
github.com/moby/sys/mountinfo v0.7.2/go.mod h1:1YOa8w8Ih7uW0wALDUgT1dTTSBrZ+HiBLGws92L2RU4=
github.com/moby/sys/reexec v0.1.0 h1:RrBi8e0EBTLEgfruBOFcxtElzRGTEUkeIFaVXgU7wok=
github.com/moby/sys/reexec v0.1.0/go.mod h1:EqjBg8F3X7iZe5pU6nRZnYCMUTXoxsjiIfHup5wYIN8=
github.com/paulmach/orb v0.13.0 h1:r7n7mQGGF+cj/CbcivEj9J3HGK+XR+yXnvzRdq9saIw=
github.com/paulmach/orb v0.13.0/go.mod h1:6scRWINywA2Jf05dcjOfLfxrUIMECvTSG2MVbRLxu/k=
github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY=
github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
github.com/pkg/profile v1.7.0 h1:hnbDkaNWPCLMO9wGLdBFTIZvzDrDfBM2072E1S9gJkA=
github.com/pkg/profile v1.7.0/go.mod h1:8Uer0jas47ZQMJ7VD+OHknK4YDY07LPUC6dEvqDjvNo=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8=
github.com/prometheus/procfs v0.20.1 h1:XwbrGOIplXW/AU3YhIhLODXMJYyC1isLFfYCsTEycfc=
github.com/prometheus/procfs v0.20.1/go.mod h1:o9EMBZGRyvDrSPH1RqdxhojkuXstoe4UlK79eF5TGGo=
github.com/rogpeppe/fastuuid v1.2.0 h1:Ppwyp6VYCF1nvBTXL3trRso7mXMlRrw9ooo375wvi2s=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
github.com/rs/zerolog v1.15.0 h1:uPRuwkWF4J6fGsJ2R0Gn2jB1EQiav9k3S6CSdygQJXY=
github.com/russross/blackfriday v1.6.0 h1:KqfZb0pUVN2lYqZUYRddxF4OR8ZMURnJIG5Y3VRLtww=
github.com/russross/blackfriday v1.6.0/go.mod h1:ti0ldHuxg49ri4ksnFxlkCfN+hvslNlmVHqNRXXJNAY=
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1 h1:lZUw3E0/J3roVtGQ+SCrUrg3ON6NgVqpn3+iol9aGu4=
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1/go.mod h1:uToXkOrWAZ6/Oc07xWQrPOhJotwFIyu2bBVN41fcDUY=
github.com/satori/go.uuid v1.2.0 h1:0uYX9dsZ2yD7q2RtLRtPSdGDWzjeM3TbMJP9utgA0ww=
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/spiffe/go-spiffe/v2 v2.6.0 h1:l+DolpxNWYgruGQVV0xsfeya3CsC7m8iBzDnMpsbLuo=
github.com/spiffe/go-spiffe/v2 v2.6.0/go.mod h1:gm2SeUoMZEtpnzPNs2Csc0D/gX33k1xIx7lEzqblHEs=
github.com/tursodatabase/libsql-client-go v0.0.0-20251219100830-236aa1ff8acc h1:lzi/5fg2EfinRlh3v//YyIhnc4tY7BTqazQGwb1ar+0=
github.com/tursodatabase/libsql-client-go v0.0.0-20251219100830-236aa1ff8acc/go.mod h1:08inkKyguB6CGGssc/JzhmQWwBgFQBgjlYFjxjRh7nU=
github.com/vertica/vertica-sql-go v1.3.6 h1:uDJPdBivsI5EwfX3NMDWZaQlVs9zTnVyxE/nYhr3bY0=
github.com/vertica/vertica-sql-go v1.3.6/go.mod h1:jnn2GFuv+O2Jcjktb7zyc4Utlbu9YVqpHH/lx63+1M4=
github.com/volatiletech/inflect v0.0.1 h1:2a6FcMQyhmPZcLa+uet3VJ8gLn/9svWhJxJYwvE8KsU=
github.com/volatiletech/inflect v0.0.1/go.mod h1:IBti31tG6phkHitLlr5j7shC5SOo//x0AjDzaJU1PLA=
github.com/volatiletech/null/v8 v8.1.2 h1:kiTiX1PpwvuugKwfvUNX/SU/5A2KGZMXfGD0DUHdKEI=
github.com/volatiletech/null/v8 v8.1.2/go.mod h1:98DbwNoKEpRrYtGjWFctievIfm4n4MxG0A6EBUcoS5g=
github.com/volatiletech/randomize v0.0.1 h1:eE5yajattWqTB2/eN8df4dw+8jwAzBtbdo5sbWC4nMk=
github.com/volatiletech/randomize v0.0.1/go.mod h1:GN3U0QYqfZ9FOJ67bzax1cqZ5q2xuj2mXrXBjWaRTlY=
github.com/volatiletech/strmangle v0.0.1 h1:UKQoHmY6be/R3tSvD2nQYrH41k43OJkidwEiC74KIzk=
github.com/volatiletech/strmangle v0.0.1/go.mod h1:F6RA6IkB5vq0yTG4GQ0UsbbRcl3ni9P76i+JrTBKFFg=
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
github.com/xdg-go/scram v1.2.0 h1:bYKF2AEwG5rqd1BumT4gAnvwU/M9nBp2pTSxeZw7Wvs=
github.com/xdg-go/scram v1.2.0/go.mod h1:3dlrS0iBaWKYVt2ZfA4cj48umJZ+cAEbR6/SjLA88I8=
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
github.com/ydb-platform/ydb-go-genproto v0.0.0-20260311095541-ebbf792c1180 h1:avIdi8eGXjKbn1WLokNR1Ofnz1k8t7tJ88YQLD/iCi8=
github.com/ydb-platform/ydb-go-genproto v0.0.0-20260311095541-ebbf792c1180/go.mod h1:Er+FePu1dNUieD+XTMDduGpQuCPssK5Q4BjF+IIXJ3I=
github.com/ydb-platform/ydb-go-sdk/v3 v3.135.0 h1:8c/M2B5W5xJd968DCedPv7DvQHuKzzhGGsO6J9x2+5U=
github.com/ydb-platform/ydb-go-sdk/v3 v3.135.0/go.mod h1:VYUUkRJkKuQPkIpgtZJj6+58Fa2g8ccAqdmaaK6HP5k=
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 h1:ilQV1hzziu+LLM3zUTJ0trRztfwgjqKnBWNtSRkbmwM=
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfSfmXjznFBSZNN13rSJjlIOI1fUNAtF7rmI=
github.com/yuin/goldmark v1.4.13 h1:fVcFKWvrslecOb/tg+Cc05dkeYx540o0FuFt3nUVDoE=
github.com/zenazn/goji v0.9.0 h1:RSQQAbXGArQ0dIDEq+PI6WqN6if+5KHu6x2Cx/GXLTQ=
github.com/ziutek/mymysql v1.5.4 h1:GB0qdRGsTwQSBVYuVShFBKaXSnSnYYC2d9knnE1LHFs=
github.com/ziutek/mymysql v1.5.4/go.mod h1:LMSpPZ6DbqWFxNCHW77HeMg9I646SAhApZ/wKdgO/C0=
go.opentelemetry.io/contrib/detectors/gcp v1.39.0 h1:kWRNZMsfBHZ+uHjiH4y7Etn2FK26LAGkNFw7RHv1DhE=
go.opentelemetry.io/contrib/detectors/gcp v1.39.0/go.mod h1:t/OGqzHBa5v6RHZwrDBJ2OirWc+4q/w2fTbLZwAKjTk=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.68.0 h1:0Qx7VGBacMm9ZENQ7TnNObTYI4ShC+lHI16seduaxZo=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.68.0/go.mod h1:Sje3i3MjSPKTSPvVWCaL8ugBzJwik3u4smCjUeuupqg=
go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk=
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=
golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU=
golang.org/x/mod v0.34.0 h1:xIHgNUUnW6sYkcM5Jleh05DvLOtwc6RitGHbDk4akRI=
golang.org/x/mod v0.34.0/go.mod h1:ykgH52iCZe79kzLLMhyCUzhMci+nQj+0XkbXpNYtVjY=
golang.org/x/oauth2 v0.34.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/oauth2 v0.35.0 h1:Mv2mzuHuZuY2+bkyWXIHMfhNdJAdwW3FuWeCPYN5GVQ=
golang.org/x/oauth2 v0.35.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
golang.org/x/time v0.11.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg=
golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg=
golang.org/x/tools v0.43.0 h1:12BdW9CeB3Z+J/I/wj34VMl8X+fEXBxVR90JeMX5E7s=
golang.org/x/tools v0.43.0/go.mod h1:uHkMso649BX2cZK6+RpuIPXS3ho2hZo4FVwfoy1vIk0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
google.golang.org/genproto/googleapis/api v0.0.0-20260120221211-b8f7ae30c516/go.mod h1:p3MLuOwURrGBRoEyFHBT3GjUwaCQVKeNqqWxlcISGdw=
gopkg.in/errgo.v2 v2.1.0 h1:0vLT13EuvQ0hNvakwLuFZ/jYrLp5F3kcWHXdRggjCE8=
gopkg.in/guregu/null.v4 v4.0.0 h1:1Wm3S1WEA2I26Kq+6vcW+w0gcDo44YKYD7YIEJNHDjg=
gopkg.in/guregu/null.v4 v4.0.0/go.mod h1:YoQhUrADuG3i9WqesrCmpNRwm1ypAgSHYqoOcTu/JrI=
gopkg.in/inconshreveable/log15.v2 v2.0.0-20180818164646-67afb5ed74ec h1:RlWgLqCMMIYYEVcAR5MDsuHlVkaIPDAF+5Dehzg8L5A=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
howett.net/plist v1.0.1 h1:37GdZ8tP09Q35o9ych3ehygcsL+HqKSwzctveSlarvM=
howett.net/plist v1.0.1/go.mod h1:lqaXoTrLY4hg8tnEzNru53gicrbv7rrk+2xJA/7hw9g=
mvdan.cc/xurls/v2 v2.6.0 h1:3NTZpeTxYVWNSokW3MKeyVkz/j7uYXYiMtXRUfmjbgI=
mvdan.cc/xurls/v2 v2.6.0/go.mod h1:bCvEZ1XvdA6wDnxY7jPPjEmigDtvtvPXAD/Exa9IMSk=
rsc.io/pdf v0.1.1 h1:k1MczvYDUvJBe93bYd7wrZLLUEcLZAuF824/I4e5Xr4=
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
+43
View File
@@ -0,0 +1,43 @@
# Multi-stage build for the R2 load harness. Mirrors backend/Dockerfile: a
# golang-alpine builder yields a static binary on distroless nonroot, with the
# dictionary DAWGs baked in from the scrabble-dictionary release (the harness runs
# the same solver as the backend, so it needs the same dictionary). The published
# scrabble-solver module is fetched from Gitea (GOPRIVATE), so the build stage needs
# git and network.
#
# The harness is not a contour service; build and run it ad hoc, from the repo root
# so go.work, pkg/, gateway/ and loadtest/ are in the Docker context:
# docker build -f loadtest/Dockerfile -t scrabble-loadtest .
# docker run --rm --network scrabble-internal -e POSTGRES_PASSWORD=... scrabble-loadtest run
# --- dictionary artifact -----------------------------------------------------
FROM alpine:3.20 AS dawg
ARG DICT_VERSION=v1.0.0
RUN apk add --no-cache curl tar
RUN mkdir -p /dawg \
&& curl -fsSL -o /tmp/dawg.tar.gz \
"https://gitea.iliadenisov.ru/developer/scrabble-dictionary/releases/download/${DICT_VERSION}/scrabble-dawg-${DICT_VERSION}.tar.gz" \
&& tar xzf /tmp/dawg.tar.gz -C /dawg
# --- build -------------------------------------------------------------------
FROM golang:1.26.3-alpine AS build
WORKDIR /src
# git: the published solver module is fetched from Gitea directly (GOPRIVATE).
RUN apk add --no-cache git
ENV GOPRIVATE=gitea.iliadenisov.ru/*
COPY go.work go.work.sum ./
COPY pkg ./pkg
COPY gateway ./gateway
COPY loadtest ./loadtest
# Reduce the workspace to what the harness needs: loadtest + gateway (edge proto) + pkg.
RUN go work edit -dropuse=./backend -dropuse=./platform/telegram
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/loadtest ./loadtest/cmd/loadtest
# --- runtime -----------------------------------------------------------------
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/loadtest /usr/local/bin/loadtest
COPY --from=dawg /dawg /opt/dawg
ENV LOADTEST_DAWG_DIR=/opt/dawg
ENTRYPOINT ["/usr/local/bin/loadtest"]
+94
View File
@@ -0,0 +1,94 @@
# loadtest — R2 stress harness
Reusable load harness for the pre-release stress pass (`PRERELEASE.md` R2/R7). It
seeds a large account population with pre-created sessions, drives virtual players
through the **gateway edge protocol** in realistic games, hammers the rate limiter,
and prints a trip-report summary. It stays in the repo for repeats.
## What it does
1. **Seed** (direct Postgres, schema `backend`): inserts `--durable` durable accounts
(each with a confirmed email identity) + `--guest` guest accounts and an active
`sessions` row per account, then hands the plaintext bearer tokens to the driver.
Token hashes match `backend/internal/session` (`hex(sha256(token))`), so the seeded
sessions resolve. Every row carries a distinctive display-name marker for cleanup.
2. **Drive** (edge protocol over h2c): assembles real 24 player games via the
invitation flow (`invitation.create``invitation.accept`, no robots), then runs
each player's turn loop — poll `game.state`, replay `game.history`, generate a legal
**mid-ranked** move with the embedded `scrabble-solver`, and `game.submit_play`
(or pass/exchange). A fraction of turns exercise nudge / chat / check-word / draft /
profile-update / stats. Each player also holds a live `Subscribe` stream. The
moderate ramp is **50 → 200 → 500** concurrent players, ~12 min per step.
3. **Hammer**: drives `games.list` from one account far above the per-user rate limit
to verify the limiter holds (`rate_limited` results) and measure its cost.
4. **Report**: per-operation latency percentiles, throughput, result-code breakdown,
live-event tally and the aggregate error rate.
The driver runs the solver **locally** because the edge protocol carries no board: the
client reconstructs it from decoded history (the same invariant as the UI).
## Connection model
The harness reaches Postgres and the gateway directly, so run it as a one-shot
container on the contour's docker network (this bypasses the host→gateway hairpin):
```sh
# from the repo root
docker build -f loadtest/Dockerfile -t scrabble-loadtest .
docker run --rm --name scrabble-loadtest --network scrabble-internal \
-e POSTGRES_PASSWORD="$TEST_POSTGRES_PASSWORD" \
scrabble-loadtest run
```
Defaults assume the contour service names: `postgres:5432` and `gateway:8081`. The
DAWGs are baked into the image (`/opt/dawg`, pinned to the dictionary release). Run with
`--name scrabble-loadtest` so the harness's own CPU/memory show up as a `scrabble-*`
series in cAdvisor (keeping it separable from the system under test). Capture the
resource baseline from the Grafana **Scrabble — Resources** dashboard
(cAdvisor + postgres_exporter) while the run is in progress.
## Commands & flags
```
loadtest run [flags] seed, drive the ramp + hammer, print the report
loadtest cleanup [flags] delete everything the harness seeded (matched by the display-name marker)
```
Key `run` flags (env in parentheses):
| flag | default | meaning |
|------|---------|---------|
| `--gateway` (`LOADTEST_GATEWAY_URL`) | `http://gateway:8081` | gateway base URL |
| `--dsn` (`LOADTEST_DSN`) | from `POSTGRES_*` | backend Postgres DSN (schema `backend`) |
| `--dawg` (`LOADTEST_DAWG_DIR`) | `/dawg` (image: `/opt/dawg`) | committed `*.dawg` directory |
| `--durable` / `--guest` | `10000` / `1000` | accounts to seed |
| `--steps` | `50,200,500` | concurrent-player ramp steps |
| `--step-dur` | `12m` | hold time per step |
| `--games-per-player` | `0` (random 35) | target concurrent games per player |
| `--tick` | `800ms` | per-player op cadence (keeps a player under the per-user limit) |
| `--secondary-prob` | `0.08` | chance per tick of a non-move op |
| `--hammer-workers` / `--hammer-dur` | `20` / `15s` | gateway-hammer (0 workers disables) |
| `--reset` / `--cleanup` | `false` | delete harness rows before / after the run |
`run` re-seeds every time (plaintext tokens are never stored), so pass `--reset` to
clear a prior run's rows first. The authoritative hard reset of the contour remains the
DB wipe (`DROP SCHEMA backend CASCADE` + backend restart).
## Build & test
```sh
go build ./loadtest/...
go vet ./loadtest/...
BACKEND_DICT_DIR=../scrabble-solver/dawg go test -count=1 ./loadtest/...
```
The DAWG-backed `moves` test runs only when `BACKEND_DICT_DIR` is set (as the engine
tests use); the pure logic (hashing, board replay, rack build, move selection, report)
runs unconditionally.
## Caveat
The harness shares the host CPU with the contour, so the early-pass resource baseline
is read with the harness's own container series in mind; a cleaner number on separate
hardware is an R7 goal. The moderate ramp keeps the generator from being the bottleneck.
+162
View File
@@ -0,0 +1,162 @@
# R2 — early stress-run trip report
The early stress pass for `PRERELEASE.md` R2. It exercises the system through the
**edge protocol** with the `scrabble/loadtest` harness, to surface logic/concurrency
bugs and capture a resource baseline that feeds R3 (edge hardening), R6 (refactor) and
R7 (final tuning). Pass bar: **diagnostic** — the run "passes" by completing without the
harness crashing; findings are recorded below, not gated.
## Method
- **Driver:** the `scrabble/loadtest` module, run as a one-shot container on the
`scrabble-internal` docker network (reaching `postgres:5432` and `gateway:8081`
directly, bypassing the host→gateway hairpin).
- **Seed:** 10 000 durable + 1 000 guest accounts with pre-created sessions written
directly to Postgres (token hash matches `backend/internal/session`), so the driver
authenticates without the per-IP-limited auth ops.
- **Games:** assembled through the real **invitation** flow (`invitation.create`
`invitation.accept`), 24 players each, no robots; variants spread over
scrabble_en / scrabble_ru / erudit_ru.
- **Play:** each virtual player holds a live `Subscribe` stream and, per tick, polls
`game.state`, replays `game.history` and submits a **mid-ranked** legal move generated
locally by the embedded `scrabble-solver` (the edge carries no board), or
passes/exchanges; a fraction exercise nudge / chat / check-word / draft / profile /
stats. A separate **gateway-hammer** floods `games.list` from one account.
- **Scale:** moderate ramp **50 → 200 → 500** concurrent players, 10 min/step (the
agreed moderate profile; harness and contour share this host's CPU).
- **Resource capture:** `docker stats` (docker API) sampled every 28 s for per-container
CPU/memory; Prometheus for edge latency/throughput, `postgres_exporter` internals and
per-service Go runtime metrics.
## Run configuration
```
loadtest run --durable 10000 --guest 1000 --steps 50,200,500 --step-dur 10m \
--tick 800ms --hammer-workers 20 --hammer-dur 15s --cleanup
```
Date: 2026-06-09. Contour: the R1-baseline schema, freshly deployed with the R2
exporters. Seeded population removed by `--cleanup` afterwards.
## Findings
### Validated (fixed within R2)
- **Harness draft payload.** `draft.save` first returned `bad_request`: the backend
draft DTO's `rack_order` is a string (the harness sent `[]`). Fixed → `ok`.
- **Harness profile marker.** `profile.update` first returned `invalid_profile`: the
editable-display-name validator (`backend/internal/account/profile.go`) forbids digits
and colons, but the seed marker was `lt:…`. Switched the marker to a distinctive
letters-only string → `ok`. Cleanup still matches it.
### By-design behaviour (correctly exercised, not bugs)
- **`chat_not_your_turn`** — chat is gated to the sender's turn
(`backend/internal/social/chat.go`); off-turn posts are correctly rejected.
- **`nudge_own_turn`** — you nudge the player whose turn it is, so a nudge on your own
turn is correctly rejected. The harness nudges/chats at random ticks, so a share of
these codes is expected.
### Observability gap (key R7 input)
- **cAdvisor yields only the root cgroup on the contour host.** Its docker factory
registers, but per-container init fails — `failed to identify the read-write layer ID
… /rootfs/var/lib/docker/image/overlayfs/…: no such file or directory` — because this
host's `/var/lib/docker` is a **separate XFS mount** not visible under cAdvisor's
`/rootfs` bind (the existing galaxy deployment on the same host has the same
limitation). So the **Scrabble — Resources** dashboard's per-container panels are empty
here, and per-container CPU/RSS for this run was captured via `docker stats` instead.
Postgres internals (`postgres_exporter`) and per-service Go runtime metrics
(`go_*` by `service_name`) work. **Recommendation for R7:** adopt the otelcol
**`docker_stats`** receiver (already the contrib image) — it reads per-container stats
via the docker API with no cgroup dependency — and/or run the final pass on hardware
where cAdvisor resolves containers. (Decision to confirm with the owner.)
### Run results
The ramp ran clean to 500 players with no harness crash, no deadlock and
`stream errors: 0`; cleanup removed all 11 000 seeded accounts (and their ~941 games).
- **Ramp:** step 1 = 50 players / 90 games, step 2 = 200 / 282, step 3 = 500 / 569.
- **Volume (30 min):** 1.20 M total edge calls, 659 req/s average. Real gameplay at
scale: **48 870 committed plays**, 52 772 `your_turn` + 159 631 `opponent_moved`
events, **2 798 games finished**.
- **Latency under load (peak, step 3):** `game.state` p50 ≈ 100 ms, p90/p99 in the
200500 ms buckets, max 849 ms; `game.submit_play` similar (p99 ≤ 500 ms, max 490 ms).
Lobby ops stayed fast (invitation/games.list p99 ≤ 10 ms).
- **Rate limiter holds.** The gateway-hammer sent 522 667 `games.list` from one account;
**522 486 (99.97 %) were `rate_limited`**, only 135 `ok` (the burst). Rejections are
cheap — p99 = 2 ms — and the gateway sustained ~16 k req/s of rejections during the
flood. The per-user limiter behaves as designed (R3 input: the cost is negligible).
**Top finding — `transport_error` under saturation.** At 500 players ~14 % of
`game.state` calls (72 429 / 519 067) and a few % of the other ops returned a Connect
`transport_error` (not a domain code). It correlates with the CPU saturation below: the
backend/gateway are pinned near one core each while the host also runs the 86 %-core
harness, so the edge sheds load (resets/timeouts) at the knee. It is **amplified by a
harness artifact** — all 500 virtual players multiplex over a *single* shared
`http2.Transport`, so 500 persistent `Subscribe` streams plus Execute calls press on one
HTTP/2 connection's concurrent-stream limit; real clients each use their own connection.
**Actions:** R7 harness — give each player (or a pool) its own transport, and run on
hardware not shared with the contour; R3 — confirm the gateway's h2c
`MaxConcurrentStreams` and edge timeouts are sized for many persistent streams.
**Minor findings:**
- `unauthenticated` on a tiny share (188 / 519 067 `game.state`, ~0.04 %) — transient
session-resolve failures under load; worth a glance in R3 but not material.
- one `internal` on `game.pass` (1 / 4 788).
- `game_finished` dominates `chat.nudge`/`chat.post` (≈ 3 900 each): the harness keeps
secondary ops on games that already ended. Harness refinement — drop finished games
from the rotation (R7).
- `nudge_own_turn` / `chat_not_your_turn` / `nudge_too_soon` are the expected turn/rate
gates, correctly exercised.
## Resource baseline
Per-container peak during step 3 (500 players), from `docker stats`:
| container | peak CPU | memory |
|-----------|---------:|-------:|
| scrabble-backend | **99 %** (~1 core) | 91 MiB |
| scrabble-gateway | **93 %** | 76 MiB |
| scrabble-postgres | **90 %** | 69 MiB |
| scrabble-loadtest (harness) | **86 %** | 42 MiB |
| scrabble-otelcol | 10 % | 110 MiB |
| scrabble-tempo | 9 % | 446 MiB |
| prometheus / postgres-exporter | ~0 % | 46 / 16 MiB |
- **The contour is CPU-bound at 500 concurrent players:** backend, gateway and Postgres
each saturate ~1 core (single-instance MVP config), so the system draws ~3 cores at
this scale; memory is modest (≤ 100 MiB per Go service). This is the sizing input for
R7 (pool sizes, GOMAXPROCS, container limits) and the prod cutover.
- **Caveat:** the harness itself peaked at **86 % of a core** on the *same host*, so the
step-3 latency and `transport_error` figures are pessimistic — the contour competed
with the generator for CPU. A clean ceiling needs separate hardware (R7).
- **Postgres:** peak 28 backend connections, ~5 581 commits/s at the peak, **100 % cache
hit ratio** (no disk reads) — the DB was comfortable; CPU, not I/O, is its limit here.
- **Goroutines:** backend 638, gateway **1 698** (it holds the 500 `Subscribe` streams +
per-request goroutines), telegram 49 — all stable, no leak across the ramp.
## Recommendations feeding later phases
- **R3 (edge hardening):** the per-user limiter holds (99.97 % rejected, p99 2 ms) — add
the per-IP body-size cap on top. Investigate the **~14 % `transport_error` on
`game.state` at 500 players**: confirm the gateway h2c `MaxConcurrentStreams` and edge
read/write timeouts are sized for many persistent `Subscribe` streams, and glance at the
~0.04 % transient `unauthenticated` resolves under load.
- **R6 (refactor):** no logic bug forced a code change beyond the two harness-payload
fixes; the run surfaced no deadlock or goroutine leak across the ramp.
- **R7 (final tuning + stress):** (1) fix the per-container observability gap — adopt the
otelcol `docker_stats` receiver so Grafana shows per-container CPU/RSS on the contour;
(2) refine the harness — per-player/pooled transports and dropping finished games from
the rotation — and run on hardware **not** shared with the contour; (3) size pools /
GOMAXPROCS / container limits from the CPU-bound peak (~1 core each for backend, gateway,
Postgres at 500 players).
## Re-running
See [`README.md`](README.md). Briefly, from the repo root:
```sh
docker build -f loadtest/Dockerfile -t scrabble-loadtest .
docker run --rm --name scrabble-loadtest --network scrabble-internal \
-e POSTGRES_PASSWORD=… scrabble-loadtest run # add --reset on a re-run
```
The harness stays in the repo for the R7 repeat.
+193
View File
@@ -0,0 +1,193 @@
// Command loadtest is the R2 reusable load harness. It seeds a large account
// population with pre-created sessions directly in the backend Postgres, then drives
// virtual players through the gateway edge protocol (real games assembled via
// invitations, legal moves generated locally by the embedded solver), and a
// gateway-hammer that verifies the rate limiter. It prints a trip-report summary.
//
// Run it as a one-shot container on the contour's docker network so it reaches
// postgres:5432 and gateway:8081 directly:
//
// docker run --rm --network scrabble-internal \
// -e POSTGRES_PASSWORD=... -v /path/to/dawg:/dawg scrabble-loadtest run
package main
import (
"context"
"errors"
"flag"
"fmt"
"log/slog"
"os"
"os/signal"
"strconv"
"strings"
"syscall"
"time"
"scrabble/loadtest/internal/edge"
"scrabble/loadtest/internal/moves"
"scrabble/loadtest/internal/report"
"scrabble/loadtest/internal/scenario"
"scrabble/loadtest/internal/seed"
)
func main() {
log := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo}))
if len(os.Args) < 2 {
usage()
os.Exit(2)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
var err error
switch os.Args[1] {
case "run":
err = cmdRun(ctx, log, os.Args[2:])
case "cleanup":
err = cmdCleanup(ctx, log, os.Args[2:])
default:
usage()
os.Exit(2)
}
if err != nil {
log.Error("loadtest failed", "cmd", os.Args[1], "err", err)
os.Exit(1)
}
}
func usage() {
fmt.Fprintln(os.Stderr, "usage: loadtest <run|cleanup> [flags]")
fmt.Fprintln(os.Stderr, " run seed accounts, drive the realistic ramp + gateway-hammer, print the report")
fmt.Fprintln(os.Stderr, " cleanup delete everything the harness seeded (matched by marker)")
}
func cmdRun(ctx context.Context, log *slog.Logger, args []string) error {
fs := flag.NewFlagSet("run", flag.ExitOnError)
gateway := fs.String("gateway", env("LOADTEST_GATEWAY_URL", "http://gateway:8081"), "gateway base URL")
dsn := fs.String("dsn", env("LOADTEST_DSN", defaultDSN()), "backend Postgres DSN")
dawgDir := fs.String("dawg", env("LOADTEST_DAWG_DIR", "/dawg"), "directory holding the committed *.dawg files")
durable := fs.Int("durable", 10000, "durable accounts to seed")
guest := fs.Int("guest", 1000, "guest accounts to seed")
stepsStr := fs.String("steps", "50,200,500", "comma-separated concurrent-player ramp steps")
stepDur := fs.Duration("step-dur", 12*time.Minute, "hold time per ramp step")
gpp := fs.Int("games-per-player", 0, "target concurrent games per player (0 => random 3..5)")
tick := fs.Duration("tick", 800*time.Millisecond, "per-player operation cadence")
secProb := fs.Float64("secondary-prob", 0.08, "chance per tick of a non-move operation")
hammerWorkers := fs.Int("hammer-workers", 20, "gateway-hammer concurrent callers (0 disables)")
hammerDur := fs.Duration("hammer-dur", 15*time.Second, "gateway-hammer duration")
reset := fs.Bool("reset", false, "delete prior harness rows before seeding")
doCleanup := fs.Bool("cleanup", false, "delete harness rows after the run")
if err := fs.Parse(args); err != nil {
return err
}
steps, err := parseSteps(*stepsStr)
if err != nil {
return err
}
reg, err := moves.Open(*dawgDir)
if err != nil {
return err
}
defer reg.Close()
sd, err := seed.New(ctx, *dsn)
if err != nil {
return err
}
defer sd.Close()
if *reset {
n, err := sd.Cleanup(ctx)
if err != nil {
return err
}
log.Info("reset", "accounts_removed", n)
}
log.Info("seeding", "durable", *durable, "guest", *guest)
pool, err := sd.Seed(ctx, *durable, *guest)
if err != nil {
return err
}
log.Info("seeded", "durable", len(pool.Durables), "guest", len(pool.Guests))
rec := report.New()
drv := scenario.NewDriver(edge.New(*gateway), reg, rec, log)
cfg := scenario.RealisticConfig{
Steps: steps, StepDur: *stepDur, GamesPerPlayer: *gpp,
Tick: *tick, SecondaryProb: *secProb,
}
if err := drv.RunRealistic(ctx, pool, cfg); err != nil && !errors.Is(err, context.Canceled) {
return err
}
if *hammerWorkers > 0 && ctx.Err() == nil && len(pool.Durables) > 0 {
drv.Hammer(ctx, pool.Durables[0], scenario.HammerConfig{Workers: *hammerWorkers, Duration: *hammerDur})
}
fmt.Println("\n==== R2 load-test report ====")
fmt.Println(rec.Summary())
if *doCleanup {
n, err := sd.Cleanup(context.WithoutCancel(ctx))
if err != nil {
return err
}
log.Info("cleanup", "accounts_removed", n)
}
return nil
}
func cmdCleanup(ctx context.Context, log *slog.Logger, args []string) error {
fs := flag.NewFlagSet("cleanup", flag.ExitOnError)
dsn := fs.String("dsn", env("LOADTEST_DSN", defaultDSN()), "backend Postgres DSN")
if err := fs.Parse(args); err != nil {
return err
}
sd, err := seed.New(ctx, *dsn)
if err != nil {
return err
}
defer sd.Close()
n, err := sd.Cleanup(ctx)
if err != nil {
return err
}
log.Info("cleanup", "accounts_removed", n)
return nil
}
// defaultDSN builds the backend Postgres DSN from the standard POSTGRES_* env the
// contour uses, pinning the backend schema.
func defaultDSN() string {
return fmt.Sprintf("postgres://%s:%s@%s:5432/%s?sslmode=disable&search_path=backend",
env("POSTGRES_USER", "scrabble"), os.Getenv("POSTGRES_PASSWORD"),
env("POSTGRES_HOST", "postgres"), env("POSTGRES_DB", "scrabble"))
}
// env returns the environment variable key or def when it is unset/empty.
func env(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
// parseSteps parses a comma-separated list of positive ramp step sizes.
func parseSteps(s string) ([]int, error) {
parts := strings.Split(s, ",")
steps := make([]int, 0, len(parts))
for _, p := range parts {
n, err := strconv.Atoi(strings.TrimSpace(p))
if err != nil || n <= 0 {
return nil, fmt.Errorf("invalid ramp steps %q", s)
}
steps = append(steps, n)
}
if len(steps) == 0 {
return nil, fmt.Errorf("no ramp steps")
}
return steps, nil
}
+16
View File
@@ -0,0 +1,16 @@
module scrabble/loadtest
go 1.26.3
require (
connectrpc.com/connect v1.19.2
gitea.iliadenisov.ru/developer/scrabble-solver v1.0.0
github.com/google/flatbuffers v23.5.26+incompatible
github.com/google/uuid v1.6.0
github.com/iliadenisov/dafsa v1.1.0
github.com/jackc/pgx/v5 v5.9.2
golang.org/x/net v0.53.0
google.golang.org/protobuf v1.36.11
scrabble/gateway v0.0.0
scrabble/pkg v0.0.0
)
+136
View File
@@ -0,0 +1,136 @@
// Package edge is the load harness's client of the gateway edge protocol: the
// Connect Execute envelope carrying FlatBuffers payloads, plus the Subscribe live
// stream, over h2c. It exposes typed wrappers for the operations the driver
// exercises, decoding responses into plain Go structs so the scenario layer never
// touches FlatBuffers directly.
package edge
import (
"context"
"crypto/tls"
"net"
"net/http"
"time"
"connectrpc.com/connect"
"golang.org/x/net/http2"
edgev1 "scrabble/gateway/proto/edge/v1"
"scrabble/gateway/proto/edge/v1/edgev1connect"
)
// Message types the driver uses, mirroring gateway/internal/transcode's catalog.
const (
msgSubmitPlay = "game.submit_play"
msgPass = "game.pass"
msgExchange = "game.exchange"
msgState = "game.state"
msgHistory = "game.history"
msgGamesList = "games.list"
msgCheckWord = "game.check_word"
msgNudge = "chat.nudge"
msgChatPost = "chat.post"
msgDraftSave = "draft.save"
msgDraftGet = "draft.get"
msgProfileGet = "profile.get"
msgProfileUpd = "profile.update"
msgStatsGet = "stats.get"
msgInvCreate = "invitation.create"
msgInvAccept = "invitation.accept"
msgInvList = "invitation.list"
msgEnqueue = "lobby.enqueue"
)
// Client speaks the edge protocol to a single gateway base URL over h2c. It is safe
// for concurrent use by many virtual players (the underlying http2.Transport pools
// and multiplexes connections).
type Client struct {
rpc edgev1connect.GatewayClient
}
// New builds a Client for baseURL (for example http://gateway:8081). The transport
// speaks HTTP/2 cleartext (h2c) to match the gateway, dialling plaintext TCP rather
// than TLS.
func New(baseURL string) *Client {
hc := &http.Client{
Transport: &http2.Transport{
AllowHTTP: true,
DialTLSContext: func(ctx context.Context, network, addr string, _ *tls.Config) (net.Conn, error) {
var d net.Dialer
return d.DialContext(ctx, network, addr)
},
},
}
return &Client{rpc: edgev1connect.NewGatewayClient(hc, baseURL)}
}
// Result is the decoded Execute envelope: Code is "ok" or a stable domain error
// code (a non-ok Code is a domain outcome, not a transport failure); Payload is the
// FlatBuffers response body (empty on error).
type Result struct {
Code string
Payload []byte
}
// execute runs one operation as token (empty for an unauthenticated op). A transport
// or connection error is returned as err; a domain rejection is reported in
// Result.Code with a nil err.
func (c *Client) execute(ctx context.Context, token, msgType string, payload []byte) (Result, error) {
req := connect.NewRequest(&edgev1.ExecuteRequest{MessageType: msgType, Payload: payload})
if token != "" {
req.Header().Set("Authorization", "Bearer "+token)
}
resp, err := c.rpc.Execute(ctx, req)
if err != nil {
return Result{Code: connectCode(err)}, err
}
return Result{Code: resp.Msg.ResultCode, Payload: resp.Msg.Payload}, nil
}
// connectCode renders a transport error as a short code for the report (e.g.
// "rate_limited" for HTTP 429, "unavailable", "deadline"), so the gateway-hammer can
// tally limiter rejections without inspecting full errors.
func connectCode(err error) string {
switch connect.CodeOf(err) {
case connect.CodeResourceExhausted:
return "rate_limited"
case connect.CodeUnauthenticated:
return "unauthenticated"
case connect.CodeUnavailable:
return "unavailable"
case connect.CodeDeadlineExceeded:
return "deadline"
default:
return "transport_error"
}
}
// Event is one decoded live event: its kind and raw FlatBuffers payload (the driver
// reacts to kind alone — your_turn / match_found drive a state fetch).
type Event struct {
Kind string
}
// Subscribe opens the live-event stream as token and invokes onEvent for each event
// until the context is cancelled or the stream ends. It blocks; run it in its own
// goroutine. Stream errors are returned for the caller to count and (optionally)
// reconnect.
func (c *Client) Subscribe(ctx context.Context, token string, onEvent func(Event)) error {
req := connect.NewRequest(&edgev1.SubscribeRequest{})
req.Header().Set("Authorization", "Bearer "+token)
stream, err := c.rpc.Subscribe(ctx, req)
if err != nil {
return err
}
defer stream.Close()
for stream.Receive() {
if onEvent != nil {
onEvent(Event{Kind: stream.Msg().Kind})
}
}
return stream.Err()
}
// pollInterval bounds how often a player re-checks one game's state; exported for the
// scenario's pacing math so a virtual player stays under the per-user rate limit.
const DefaultPollInterval = 3 * time.Second
+186
View File
@@ -0,0 +1,186 @@
package edge
import (
fb "scrabble/pkg/fbs/scrabblefb"
)
// Game is the decoded non-private game summary the driver needs to decide a turn.
type Game struct {
ID string
Variant string
DictVer string
Status string
Players int
ToMove int
MoveCount int
Seats []string // account ids in seat order
}
// Active reports whether the game is still in progress.
func (g Game) Active() bool { return g.Status == "active" }
// SeatOf returns the seat index of accountID, or -1 if it is not seated.
func (g Game) SeatOf(accountID string) int {
for i, id := range g.Seats {
if id == accountID {
return i
}
}
return -1
}
// State is a player's private view: the shared game plus their seat, rack (alphabet
// indices; 255 a blank) and bag size.
type State struct {
Game Game
Seat int
Rack []byte
BagLen int
}
// Tile is one placed tile from a decoded history record (concrete letter, blank flag).
type Tile struct {
Row, Col int
Letter string
Blank bool
}
// Move is one decoded history record (a committed play carries Tiles; pass/exchange
// carry only Action).
type Move struct {
Action string
Dir string
Tiles []Tile
}
// Invitation is the decoded subset the assembler matches on.
type Invitation struct {
ID string
InviterID string
Status string
GameID string
}
func decodeGameView(gv *fb.GameView) Game {
g := Game{
ID: string(gv.Id()),
Variant: string(gv.Variant()),
DictVer: string(gv.DictVersion()),
Status: string(gv.Status()),
Players: int(gv.Players()),
ToMove: int(gv.ToMove()),
MoveCount: int(gv.MoveCount()),
}
n := gv.SeatsLength()
g.Seats = make([]string, n)
var sv fb.SeatView
for j := 0; j < n; j++ {
if gv.Seats(&sv, j) {
g.Seats[sv.Seat()] = string(sv.AccountId())
}
}
return g
}
// decodeState reads a StateView payload.
func decodeState(payload []byte) State {
sv := fb.GetRootAsStateView(payload, 0)
var gv fb.GameView
st := State{
Seat: int(sv.Seat()),
BagLen: int(sv.BagLen()),
Rack: append([]byte(nil), sv.RackBytes()...),
}
if g := sv.Game(&gv); g != nil {
st.Game = decodeGameView(g)
}
return st
}
// decodeHistory reads a History payload into the decoded move journal.
func decodeHistory(payload []byte) []Move {
h := fb.GetRootAsHistory(payload, 0)
n := h.MovesLength()
moves := make([]Move, 0, n)
var mr fb.MoveRecord
for j := 0; j < n; j++ {
if !h.Moves(&mr, j) {
continue
}
m := Move{Action: string(mr.Action()), Dir: string(mr.Dir())}
tn := mr.TilesLength()
m.Tiles = make([]Tile, 0, tn)
var tr fb.TileRecord
for k := 0; k < tn; k++ {
if mr.Tiles(&tr, k) {
m.Tiles = append(m.Tiles, Tile{
Row: int(tr.Row()), Col: int(tr.Col()),
Letter: string(tr.Letter()), Blank: tr.Blank(),
})
}
}
moves = append(moves, m)
}
return moves
}
// decodeMoveResultGame reads a MoveResult payload and returns its post-move game.
func decodeMoveResultGame(payload []byte) Game {
mr := fb.GetRootAsMoveResult(payload, 0)
var gv fb.GameView
if g := mr.Game(&gv); g != nil {
return decodeGameView(g)
}
return Game{}
}
// decodeGameList reads a GameList payload.
func decodeGameList(payload []byte) []Game {
gl := fb.GetRootAsGameList(payload, 0)
n := gl.GamesLength()
games := make([]Game, 0, n)
var gv fb.GameView
for j := 0; j < n; j++ {
if gl.Games(&gv, j) {
games = append(games, decodeGameView(&gv))
}
}
return games
}
// decodeInvitationList reads an InvitationList payload into the matched subset.
func decodeInvitationList(payload []byte) []Invitation {
il := fb.GetRootAsInvitationList(payload, 0)
n := il.InvitationsLength()
out := make([]Invitation, 0, n)
var inv fb.Invitation
var ref fb.AccountRef
for j := 0; j < n; j++ {
if !il.Invitations(&inv, j) {
continue
}
iv := Invitation{
ID: string(inv.Id()),
Status: string(inv.Status()),
GameID: string(inv.GameId()),
}
if r := inv.Inviter(&ref); r != nil {
iv.InviterID = string(r.AccountId())
}
out = append(out, iv)
}
return out
}
// decodeMatch reads a MatchResult payload.
func decodeMatch(payload []byte) (matched bool, game Game) {
mr := fb.GetRootAsMatchResult(payload, 0)
if !mr.Matched() {
return false, Game{}
}
var gv fb.GameView
if g := mr.Game(&gv); g != nil {
return true, decodeGameView(g)
}
return true, Game{}
}
+184
View File
@@ -0,0 +1,184 @@
package edge
import (
flatbuffers "github.com/google/flatbuffers/go"
fb "scrabble/pkg/fbs/scrabblefb"
)
// PlayTile is one tile to place, addressed by alphabet index (255 marks a blank's
// carrier letter together with Blank=true), as the submit-play request carries it.
type PlayTile struct {
Row, Col int
Letter byte
Blank bool
}
// gameAction builds a GameActionRequest payload (just a game id): pass, nudge,
// history, draft.get.
func gameAction(gameID string) []byte {
b := flatbuffers.NewBuilder(64)
gid := b.CreateString(gameID)
fb.GameActionRequestStart(b)
fb.GameActionRequestAddGameId(b, gid)
b.Finish(fb.GameActionRequestEnd(b))
return b.FinishedBytes()
}
// stateReq builds a StateRequest payload. includeAlphabet asks the backend to embed
// the variant alphabet table (the driver sets it once per variant).
func stateReq(gameID string, includeAlphabet bool) []byte {
b := flatbuffers.NewBuilder(64)
gid := b.CreateString(gameID)
fb.StateRequestStart(b)
fb.StateRequestAddGameId(b, gid)
fb.StateRequestAddIncludeAlphabet(b, includeAlphabet)
b.Finish(fb.StateRequestEnd(b))
return b.FinishedBytes()
}
// submitPlay builds a SubmitPlayRequest payload. dir is "H" or "V"; tiles are the
// newly-placed tiles in main-word order.
func submitPlay(gameID, dir string, tiles []PlayTile) []byte {
b := flatbuffers.NewBuilder(256)
gid := b.CreateString(gameID)
d := b.CreateString(dir)
offs := make([]flatbuffers.UOffsetT, len(tiles))
for i, t := range tiles {
fb.PlayTileStart(b)
fb.PlayTileAddRow(b, int32(t.Row))
fb.PlayTileAddCol(b, int32(t.Col))
fb.PlayTileAddLetter(b, t.Letter)
fb.PlayTileAddBlank(b, t.Blank)
offs[i] = fb.PlayTileEnd(b)
}
fb.SubmitPlayRequestStartTilesVector(b, len(offs))
for i := len(offs) - 1; i >= 0; i-- {
b.PrependUOffsetT(offs[i])
}
tilesVec := b.EndVector(len(offs))
fb.SubmitPlayRequestStart(b)
fb.SubmitPlayRequestAddGameId(b, gid)
fb.SubmitPlayRequestAddDir(b, d)
fb.SubmitPlayRequestAddTiles(b, tilesVec)
b.Finish(fb.SubmitPlayRequestEnd(b))
return b.FinishedBytes()
}
// exchange builds an ExchangeRequest payload swapping the listed rack tiles (alphabet
// indices; 255 a blank).
func exchange(gameID string, tiles []byte) []byte {
b := flatbuffers.NewBuilder(64)
gid := b.CreateString(gameID)
vec := b.CreateByteVector(tiles)
fb.ExchangeRequestStart(b)
fb.ExchangeRequestAddGameId(b, gid)
fb.ExchangeRequestAddTiles(b, vec)
b.Finish(fb.ExchangeRequestEnd(b))
return b.FinishedBytes()
}
// checkWord builds a CheckWordRequest payload (alphabet indices for the word).
func checkWord(gameID string, word []byte) []byte {
b := flatbuffers.NewBuilder(64)
gid := b.CreateString(gameID)
vec := b.CreateByteVector(word)
fb.CheckWordRequestStart(b)
fb.CheckWordRequestAddGameId(b, gid)
fb.CheckWordRequestAddWord(b, vec)
b.Finish(fb.CheckWordRequestEnd(b))
return b.FinishedBytes()
}
// chatPost builds a ChatPostRequest payload.
func chatPost(gameID, body string) []byte {
b := flatbuffers.NewBuilder(128)
gid := b.CreateString(gameID)
bd := b.CreateString(body)
fb.ChatPostRequestStart(b)
fb.ChatPostRequestAddGameId(b, gid)
fb.ChatPostRequestAddBody(b, bd)
b.Finish(fb.ChatPostRequestEnd(b))
return b.FinishedBytes()
}
// draftSave builds a DraftRequest payload carrying the opaque composition JSON.
func draftSave(gameID, jsonStr string) []byte {
b := flatbuffers.NewBuilder(128)
gid := b.CreateString(gameID)
j := b.CreateString(jsonStr)
fb.DraftRequestStart(b)
fb.DraftRequestAddGameId(b, gid)
fb.DraftRequestAddJson(b, j)
b.Finish(fb.DraftRequestEnd(b))
return b.FinishedBytes()
}
// updateProfile builds an UpdateProfileRequest payload. It resends the marker display
// name and sane defaults so the account stays findable by the seeder's Cleanup.
func updateProfile(displayName, lang string) []byte {
b := flatbuffers.NewBuilder(192)
name := b.CreateString(displayName)
pl := b.CreateString(lang)
tz := b.CreateString("UTC")
as := b.CreateString("00:00")
ae := b.CreateString("07:00")
fb.UpdateProfileRequestStart(b)
fb.UpdateProfileRequestAddDisplayName(b, name)
fb.UpdateProfileRequestAddPreferredLanguage(b, pl)
fb.UpdateProfileRequestAddTimeZone(b, tz)
fb.UpdateProfileRequestAddAwayStart(b, as)
fb.UpdateProfileRequestAddAwayEnd(b, ae)
fb.UpdateProfileRequestAddBlockChat(b, false)
fb.UpdateProfileRequestAddBlockFriendRequests(b, false)
fb.UpdateProfileRequestAddNotificationsInAppOnly(b, true)
b.Finish(fb.UpdateProfileRequestEnd(b))
return b.FinishedBytes()
}
// createInvitation builds a CreateInvitationRequest payload. turnTimeoutSecs 0 asks
// the backend for its default; dropoutTiles "remove" is the standard policy.
func createInvitation(inviteeIDs []string, variant string, turnTimeoutSecs int) []byte {
b := flatbuffers.NewBuilder(256)
idOffs := make([]flatbuffers.UOffsetT, len(inviteeIDs))
for i, id := range inviteeIDs {
idOffs[i] = b.CreateString(id)
}
fb.CreateInvitationRequestStartInviteeIdsVector(b, len(idOffs))
for i := len(idOffs) - 1; i >= 0; i-- {
b.PrependUOffsetT(idOffs[i])
}
ids := b.EndVector(len(idOffs))
variantOff := b.CreateString(variant)
dropout := b.CreateString("remove")
fb.CreateInvitationRequestStart(b)
fb.CreateInvitationRequestAddInviteeIds(b, ids)
fb.CreateInvitationRequestAddVariant(b, variantOff)
fb.CreateInvitationRequestAddTurnTimeoutSecs(b, int32(turnTimeoutSecs))
fb.CreateInvitationRequestAddHintsAllowed(b, true)
fb.CreateInvitationRequestAddHintsPerPlayer(b, 1)
fb.CreateInvitationRequestAddDropoutTiles(b, dropout)
b.Finish(fb.CreateInvitationRequestEnd(b))
return b.FinishedBytes()
}
// invitationAction builds an InvitationActionRequest payload (accept / decline /
// cancel by id).
func invitationAction(invitationID string) []byte {
b := flatbuffers.NewBuilder(64)
id := b.CreateString(invitationID)
fb.InvitationActionRequestStart(b)
fb.InvitationActionRequestAddInvitationId(b, id)
b.Finish(fb.InvitationActionRequestEnd(b))
return b.FinishedBytes()
}
// enqueueReq builds an EnqueueRequest payload (join the per-variant auto-match pool).
func enqueueReq(variant string) []byte {
b := flatbuffers.NewBuilder(64)
v := b.CreateString(variant)
fb.EnqueueRequestStart(b)
fb.EnqueueRequestAddVariant(b, v)
b.Finish(fb.EnqueueRequestEnd(b))
return b.FinishedBytes()
}
+136
View File
@@ -0,0 +1,136 @@
package edge
import "context"
// The typed operations below each build a request, run Execute and decode the
// response. They return the decoded value (where any), the domain result code
// ("ok" or a stable error code) and a transport error. The scenario layer times the
// call and records the code; a non-"ok" code with a nil error is a domain rejection
// (for example "not_your_turn"), not a failure of the harness.
// State fetches the caller's private view of a game.
func (c *Client) State(ctx context.Context, token, gameID string) (State, string, error) {
r, err := c.execute(ctx, token, msgState, stateReq(gameID, false))
if err != nil || r.Code != "ok" {
return State{}, r.Code, err
}
return decodeState(r.Payload), r.Code, nil
}
// History fetches a game's decoded move journal (the board-replay source).
func (c *Client) History(ctx context.Context, token, gameID string) ([]Move, string, error) {
r, err := c.execute(ctx, token, msgHistory, gameAction(gameID))
if err != nil || r.Code != "ok" {
return nil, r.Code, err
}
return decodeHistory(r.Payload), r.Code, nil
}
// SubmitPlay commits a play and returns the post-move game.
func (c *Client) SubmitPlay(ctx context.Context, token, gameID, dir string, tiles []PlayTile) (Game, string, error) {
r, err := c.execute(ctx, token, msgSubmitPlay, submitPlay(gameID, dir, tiles))
if err != nil || r.Code != "ok" {
return Game{}, r.Code, err
}
return decodeMoveResultGame(r.Payload), r.Code, nil
}
// Pass forfeits the turn and returns the post-move game.
func (c *Client) Pass(ctx context.Context, token, gameID string) (Game, string, error) {
r, err := c.execute(ctx, token, msgPass, gameAction(gameID))
if err != nil || r.Code != "ok" {
return Game{}, r.Code, err
}
return decodeMoveResultGame(r.Payload), r.Code, nil
}
// Exchange swaps the listed rack tiles and returns the post-move game.
func (c *Client) Exchange(ctx context.Context, token, gameID string, tiles []byte) (Game, string, error) {
r, err := c.execute(ctx, token, msgExchange, exchange(gameID, tiles))
if err != nil || r.Code != "ok" {
return Game{}, r.Code, err
}
return decodeMoveResultGame(r.Payload), r.Code, nil
}
// Nudge prods the opponent whose turn it is.
func (c *Client) Nudge(ctx context.Context, token, gameID string) (string, error) {
r, err := c.execute(ctx, token, msgNudge, gameAction(gameID))
return r.Code, err
}
// ChatPost posts a per-game chat line.
func (c *Client) ChatPost(ctx context.Context, token, gameID, body string) (string, error) {
r, err := c.execute(ctx, token, msgChatPost, chatPost(gameID, body))
return r.Code, err
}
// CheckWord looks a word up in the game's pinned dictionary.
func (c *Client) CheckWord(ctx context.Context, token, gameID string, word []byte) (string, error) {
r, err := c.execute(ctx, token, msgCheckWord, checkWord(gameID, word))
return r.Code, err
}
// DraftSave stores the player's client-side composition.
func (c *Client) DraftSave(ctx context.Context, token, gameID, jsonStr string) (string, error) {
r, err := c.execute(ctx, token, msgDraftSave, draftSave(gameID, jsonStr))
return r.Code, err
}
// DraftGet retrieves the player's stored composition.
func (c *Client) DraftGet(ctx context.Context, token, gameID string) (string, error) {
r, err := c.execute(ctx, token, msgDraftGet, gameAction(gameID))
return r.Code, err
}
// ProfileUpdate overwrites the profile, resending the marker display name.
func (c *Client) ProfileUpdate(ctx context.Context, token, displayName, lang string) (string, error) {
r, err := c.execute(ctx, token, msgProfileUpd, updateProfile(displayName, lang))
return r.Code, err
}
// Stats reads the caller's lifetime statistics.
func (c *Client) Stats(ctx context.Context, token string) (string, error) {
r, err := c.execute(ctx, token, msgStatsGet, nil)
return r.Code, err
}
// GamesList lists the caller's games (active and finished).
func (c *Client) GamesList(ctx context.Context, token string) ([]Game, string, error) {
r, err := c.execute(ctx, token, msgGamesList, nil)
if err != nil || r.Code != "ok" {
return nil, r.Code, err
}
return decodeGameList(r.Payload), r.Code, nil
}
// CreateInvitation proposes a 2-4 player friend game to the named invitees.
func (c *Client) CreateInvitation(ctx context.Context, token string, inviteeIDs []string, variant string) (string, error) {
r, err := c.execute(ctx, token, msgInvCreate, createInvitation(inviteeIDs, variant, 0))
return r.Code, err
}
// AcceptInvitation accepts an invitation by id (the completing accept starts the game).
func (c *Client) AcceptInvitation(ctx context.Context, token, invitationID string) (string, error) {
r, err := c.execute(ctx, token, msgInvAccept, invitationAction(invitationID))
return r.Code, err
}
// ListInvitations lists the caller's open invitations.
func (c *Client) ListInvitations(ctx context.Context, token string) ([]Invitation, string, error) {
r, err := c.execute(ctx, token, msgInvList, nil)
if err != nil || r.Code != "ok" {
return nil, r.Code, err
}
return decodeInvitationList(r.Payload), r.Code, nil
}
// Enqueue joins the per-variant auto-match pool and reports any immediate pairing.
func (c *Client) Enqueue(ctx context.Context, token, variant string) (bool, Game, string, error) {
r, err := c.execute(ctx, token, msgEnqueue, enqueueReq(variant))
if err != nil || r.Code != "ok" {
return false, Game{}, r.Code, err
}
matched, game := decodeMatch(r.Payload)
return matched, game, r.Code, nil
}
+186
View File
@@ -0,0 +1,186 @@
// Package moves turns a game's public history and the caller's private rack into a
// legal turn, by reconstructing the board and running the embedded scrabble-solver
// locally (the edge protocol carries no board — the client replays history). It
// picks a mid-ranked move so games progress realistically rather than optimally.
package moves
import (
"fmt"
"math/rand"
"path/filepath"
"gitea.iliadenisov.ru/developer/scrabble-solver/board"
"gitea.iliadenisov.ru/developer/scrabble-solver/rack"
"gitea.iliadenisov.ru/developer/scrabble-solver/rules"
"gitea.iliadenisov.ru/developer/scrabble-solver/scrabble"
dawg "github.com/iliadenisov/dafsa"
"scrabble/loadtest/internal/edge"
)
// blankIndex is the rack/exchange sentinel for a blank tile on the wire (Stage 13).
const blankIndex = 255
// variantSpec maps an edge variant label to its ruleset constructor and committed
// DAWG filename (the descriptive names kept by R1).
type variantSpec struct {
ruleset func() *rules.Ruleset
dawg string
}
var specs = map[string]variantSpec{
"scrabble_en": {rules.English, "en_sowpods.dawg"},
"scrabble_ru": {rules.RussianScrabble, "ru_scrabble.dawg"},
"erudit_ru": {rules.Erudit, "ru_erudit.dawg"},
}
// Variants returns the edge variant labels the harness drives, in catalogue order.
func Variants() []string { return []string{"scrabble_en", "scrabble_ru", "erudit_ru"} }
// engine is one loaded variant: its ruleset and a solver over its DAWG.
type engine struct {
rs *rules.Ruleset
finder dawg.Finder
solver *scrabble.Solver
}
// Registry holds a solver per variant, built from the committed DAWGs in dir. It is
// safe for concurrent use: every Pick builds its own board and rack, and the solver
// holds only read-only state (the same way the backend shares one solver per variant
// across concurrent games).
type Registry struct {
engines map[string]*engine
}
// Open loads every variant's DAWG from dir and builds a solver over each. dir holds
// the committed dawg files (the sibling scrabble-solver checkout's dawg/, or the
// dictionary release artifact).
func Open(dir string) (*Registry, error) {
r := &Registry{engines: make(map[string]*engine)}
for label, spec := range specs {
rs := spec.ruleset()
finder, err := dawg.Load(filepath.Join(dir, spec.dawg))
if err != nil {
r.Close()
return nil, fmt.Errorf("moves: load %s dawg %s from %s: %w", label, spec.dawg, dir, err)
}
r.engines[label] = &engine{rs: rs, finder: finder, solver: scrabble.NewSolver(rs, finder)}
}
return r, nil
}
// Close releases every loaded DAWG.
func (r *Registry) Close() {
for _, e := range r.engines {
if e.finder != nil {
_ = e.finder.Close()
}
}
}
// Action is a chosen turn. Kind is "play", "exchange" or "pass". A play carries Dir
// ("H"/"V") and Tiles; an exchange carries Exchange (rack indices to swap).
type Action struct {
Kind string
Dir string
Tiles []edge.PlayTile
Exchange []byte
}
// Pick reconstructs the board for variant from history, builds the rack from the
// alphabet-index rack, generates the legal plays and returns a mid-ranked one. With
// no legal play it exchanges (when the bag holds a full rack) or passes. rng makes
// the choice deterministic per caller; pass each virtual player its own *rand.Rand
// (rand.Rand is not safe for concurrent use).
func (r *Registry) Pick(variant string, history []edge.Move, rackIdx []byte, bagLen int, rng *rand.Rand) (Action, error) {
e, ok := r.engines[variant]
if !ok {
return Action{}, fmt.Errorf("moves: unknown variant %q", variant)
}
b, err := replayBoard(e.rs, history)
if err != nil {
return Action{}, err
}
legal := e.solver.GenerateMoves(b, buildRack(e.rs, rackIdx), scrabble.Both)
if len(legal) == 0 {
return noPlay(rackIdx, bagLen >= e.rs.RackSize), nil
}
m := midRanked(legal, rng)
return Action{Kind: "play", Dir: dirString(m.Dir), Tiles: toPlayTiles(m.Tiles)}, nil
}
// toPlayTiles maps the solver's newly-placed tiles to the edge submit-play tiles
// (addressed by alphabet index, carrying the blank flag).
func toPlayTiles(placements []scrabble.Placement) []edge.PlayTile {
tiles := make([]edge.PlayTile, len(placements))
for i, p := range placements {
tiles[i] = edge.PlayTile{Row: p.Row, Col: p.Col, Letter: p.Letter, Blank: p.Blank}
}
return tiles
}
// replayBoard mirrors backend engine.ReplayBoard using only the solver's public API:
// each play record's letters are re-indexed through the alphabet and applied to an
// empty board. Non-play records are ignored.
func replayBoard(rs *rules.Ruleset, history []edge.Move) (*board.Board, error) {
b := board.New(rs.Rows, rs.Cols)
for _, rec := range history {
if rec.Action != "play" {
continue
}
ps := make([]scrabble.Placement, len(rec.Tiles))
for i, t := range rec.Tiles {
idx, err := rs.Alphabet.Index(t.Letter)
if err != nil {
return nil, fmt.Errorf("moves: replay letter %q at (%d,%d): %w", t.Letter, t.Row, t.Col, err)
}
ps[i] = scrabble.Placement{Row: t.Row, Col: t.Col, Letter: idx, Blank: t.Blank}
}
scrabble.Apply(b, scrabble.Move{Tiles: ps})
}
return b, nil
}
// buildRack turns the alphabet-index rack (255 a blank) into a solver Rack.
func buildRack(rs *rules.Ruleset, rackIdx []byte) rack.Rack {
rk := rack.New(rs.Alphabet.Size())
for _, idx := range rackIdx {
if idx == blankIndex {
rk.AddBlank()
} else {
rk.Add(idx)
}
}
return rk
}
// midRanked returns a move from the middle third of the score-ranked list
// (GenerateMoves returns highest-first), spreading the pick within that band with
// rng. A tiny list yields its lowest-scoring move.
func midRanked(moves []scrabble.Move, rng *rand.Rand) scrabble.Move {
n := len(moves)
if n <= 2 {
return moves[n-1]
}
lo, hi := n/3, 2*n/3
if hi <= lo {
hi = lo + 1
}
return moves[lo+rng.Intn(hi-lo)]
}
// noPlay chooses an exchange (when the bag can refill a full rack) or a pass.
func noPlay(rackIdx []byte, canExchange bool) Action {
if canExchange && len(rackIdx) > 0 {
return Action{Kind: "exchange", Exchange: append([]byte(nil), rackIdx...)}
}
return Action{Kind: "pass"}
}
// dirString renders a solver direction as the "H"/"V" the edge submit-play expects.
func dirString(d scrabble.Direction) string {
if d == scrabble.Vertical {
return "V"
}
return "H"
}
+157
View File
@@ -0,0 +1,157 @@
package moves
import (
"math/rand"
"os"
"strings"
"testing"
"gitea.iliadenisov.ru/developer/scrabble-solver/board"
"gitea.iliadenisov.ru/developer/scrabble-solver/rules"
"gitea.iliadenisov.ru/developer/scrabble-solver/scrabble"
"scrabble/loadtest/internal/edge"
)
// TestReplayBoardMatchesParse checks that replaying decoded history reproduces the
// exact board (positions, letters and blank flags) that board.Parse builds from the
// equivalent text grid, and that non-play records are ignored.
func TestReplayBoardMatchesParse(t *testing.T) {
rs := rules.English()
history := []edge.Move{
{Action: "pass"}, // must be ignored
{Action: "play", Tiles: []edge.Tile{
{Row: 7, Col: 7, Letter: "c"},
{Row: 7, Col: 8, Letter: "a"},
{Row: 7, Col: 9, Letter: "t"},
}},
{Action: "play", Tiles: []edge.Tile{
{Row: 7, Col: 10, Letter: "s", Blank: true}, // a blank standing for s
}},
}
got, err := replayBoard(rs, history)
if err != nil {
t.Fatalf("replayBoard: %v", err)
}
rows := make([]string, rs.Rows)
for i := range rows {
rows[i] = strings.Repeat(".", rs.Cols)
}
// row 7: cols 0-6 empty, cat at 7-9, an uppercase S (blank) at 10.
rows[7] = strings.Repeat(".", 7) + "cat" + "S" + strings.Repeat(".", rs.Cols-11)
want, err := board.Parse(rows, rs.Alphabet)
if err != nil {
t.Fatalf("board.Parse: %v", err)
}
for r := 0; r < rs.Rows; r++ {
for c := 0; c < rs.Cols; c++ {
if got.At(r, c) != want.At(r, c) {
t.Fatalf("cell (%d,%d): replay = %#x, parse = %#x", r, c, got.At(r, c), want.At(r, c))
}
}
}
}
// TestBuildRack checks the alphabet-index rack (255 a blank) is reconstructed faithfully.
func TestBuildRack(t *testing.T) {
rs := rules.English()
rk := buildRack(rs, []byte{0, 0, 2, blankIndex}) // a a c blank
if rk.Count(0) != 2 {
t.Errorf("count(a) = %d, want 2", rk.Count(0))
}
if rk.Count(2) != 1 {
t.Errorf("count(c) = %d, want 1", rk.Count(2))
}
if rk.Blanks() != 1 {
t.Errorf("blanks = %d, want 1", rk.Blanks())
}
if rk.Total() != 4 {
t.Errorf("total = %d, want 4", rk.Total())
}
}
// TestMidRanked checks the pick always lands in the middle third of a ranked list and
// that tiny lists yield their lowest-scoring move.
func TestMidRanked(t *testing.T) {
ms := make([]scrabble.Move, 9) // scores 100..92, index i has score 100-i
for i := range ms {
ms[i] = scrabble.Move{Score: 100 - i}
}
rng := rand.New(rand.NewSource(1))
for n := 0; n < 100; n++ {
idx := 100 - midRanked(ms, rng).Score // recover the index from the score
if idx < 3 || idx >= 6 {
t.Fatalf("picked index %d outside middle third [3,6)", idx)
}
}
if got := midRanked([]scrabble.Move{{Score: 5}}, rng).Score; got != 5 {
t.Errorf("n=1 pick score = %d, want 5", got)
}
if got := midRanked([]scrabble.Move{{Score: 9}, {Score: 4}}, rng).Score; got != 4 {
t.Errorf("n=2 pick score = %d, want 4 (lower-scoring)", got)
}
}
// TestToPlayTiles checks the solver-placement to edge-tile mapping, including blanks.
func TestToPlayTiles(t *testing.T) {
tiles := toPlayTiles([]scrabble.Placement{
{Row: 1, Col: 2, Letter: 5},
{Row: 1, Col: 3, Letter: 255, Blank: true},
})
want := []edge.PlayTile{
{Row: 1, Col: 2, Letter: 5},
{Row: 1, Col: 3, Letter: 255, Blank: true},
}
if len(tiles) != len(want) {
t.Fatalf("len = %d, want %d", len(tiles), len(want))
}
for i := range want {
if tiles[i] != want[i] {
t.Errorf("tile %d = %+v, want %+v", i, tiles[i], want[i])
}
}
}
// TestPickUnknownVariant rejects a variant the registry does not hold.
func TestPickUnknownVariant(t *testing.T) {
reg := &Registry{engines: map[string]*engine{}}
if _, err := reg.Pick("nope", nil, nil, 0, rand.New(rand.NewSource(1))); err == nil {
t.Fatal("want error for an unknown variant")
}
}
// TestPickWithDawg drives the full path against the committed DAWGs when they are
// available (BACKEND_DICT_DIR, as the engine tests use); it generates a first-move
// play from a productive rack.
func TestPickWithDawg(t *testing.T) {
dir := os.Getenv("BACKEND_DICT_DIR")
if dir == "" {
t.Skip("BACKEND_DICT_DIR not set; skipping DAWG-backed test")
}
reg, err := Open(dir)
if err != nil {
t.Fatalf("Open(%s): %v", dir, err)
}
defer reg.Close()
rng := rand.New(rand.NewSource(1))
rack := []byte{2, 0, 19, 18, 4, 17, 13} // c a t s e r n — a productive English rack
act, err := reg.Pick("scrabble_en", nil, rack, 90, rng)
if err != nil {
t.Fatalf("Pick: %v", err)
}
switch act.Kind {
case "play":
if len(act.Tiles) == 0 {
t.Error("play action has no tiles")
}
if act.Dir != "H" && act.Dir != "V" {
t.Errorf("dir = %q, want H or V", act.Dir)
}
case "exchange", "pass":
// acceptable when the rack has no legal first move
default:
t.Errorf("unexpected action kind %q", act.Kind)
}
}
+204
View File
@@ -0,0 +1,204 @@
// Package report collects per-operation latency, result-code and live-event counts
// across all virtual players and renders a text summary for the R2 trip report. It
// is safe for concurrent use. Latencies go into fixed buckets (a Prometheus-style
// histogram) so percentiles cost no per-sample memory at load-test scale.
package report
import (
"fmt"
"sort"
"strings"
"sync"
"time"
)
// boundsMs are the upper bounds (milliseconds) of the latency histogram buckets; a
// trailing overflow bucket catches anything slower.
var boundsMs = []float64{1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000}
type opStat struct {
count int64
sumNs int64
maxNs int64
codes map[string]int64
buckets []int64 // len(boundsMs)+1; last is the overflow bucket
}
func newOpStat() *opStat {
return &opStat{codes: map[string]int64{}, buckets: make([]int64, len(boundsMs)+1)}
}
func (s *opStat) record(code string, d time.Duration) {
s.count++
s.sumNs += int64(d)
if int64(d) > s.maxNs {
s.maxNs = int64(d)
}
s.codes[code]++
ms := float64(d) / float64(time.Millisecond)
i := sort.SearchFloat64s(boundsMs, ms)
s.buckets[i]++
}
// quantile estimates the q-th percentile (0<q<1) as the upper bound of the bucket
// the q-th sample falls in; the overflow bucket renders as ">5000".
func (s *opStat) quantile(q float64) string {
if s.count == 0 {
return "-"
}
target := int64(q*float64(s.count) + 0.5)
if target < 1 {
target = 1
}
var cum int64
for i, n := range s.buckets {
cum += n
if cum >= target {
if i == len(boundsMs) {
return ">5000"
}
return fmt.Sprintf("%g", boundsMs[i])
}
}
return ">5000"
}
// Recorder accumulates the run's measurements.
type Recorder struct {
mu sync.Mutex
ops map[string]*opStat
events map[string]int64
streamErrs int64
start time.Time
}
// New returns an empty Recorder with the run clock started.
func New() *Recorder {
return &Recorder{ops: map[string]*opStat{}, events: map[string]int64{}, start: time.Now()}
}
// Record logs one operation call: its name, domain/transport code and latency.
func (r *Recorder) Record(op, code string, d time.Duration) {
r.mu.Lock()
defer r.mu.Unlock()
s := r.ops[op]
if s == nil {
s = newOpStat()
r.ops[op] = s
}
s.record(code, d)
}
// Event logs one received live event of the given kind.
func (r *Recorder) Event(kind string) {
r.mu.Lock()
defer r.mu.Unlock()
r.events[kind]++
}
// StreamErr logs one Subscribe stream error (a drop the player reconnects from).
func (r *Recorder) StreamErr() {
r.mu.Lock()
defer r.mu.Unlock()
r.streamErrs++
}
// Totals returns the aggregate call count and the count of non-"ok" results, for the
// pass/fail summary.
func (r *Recorder) Totals() (calls, nonOK int64) {
r.mu.Lock()
defer r.mu.Unlock()
for _, s := range r.ops {
calls += s.count
for code, n := range s.codes {
if code != "ok" {
nonOK += n
}
}
}
return calls, nonOK
}
// Summary renders the human-readable run report: a per-operation table (count,
// throughput, p50/p90/p99/max latency, code breakdown), the live-event tally and the
// aggregate error rate.
func (r *Recorder) Summary() string {
r.mu.Lock()
defer r.mu.Unlock()
elapsed := time.Since(r.start).Seconds()
if elapsed <= 0 {
elapsed = 1
}
names := make([]string, 0, len(r.ops))
for op := range r.ops {
names = append(names, op)
}
sort.Strings(names)
var b strings.Builder
fmt.Fprintf(&b, "elapsed: %.0fs\n\n", elapsed)
fmt.Fprintf(&b, "%-20s %8s %8s %7s %7s %7s %7s %s\n",
"operation", "count", "req/s", "p50ms", "p90ms", "p99ms", "maxms", "codes")
var totalCalls, totalNonOK int64
for _, op := range names {
s := r.ops[op]
totalCalls += s.count
var nonOK int64
for code, n := range s.codes {
if code != "ok" {
nonOK += n
}
}
totalNonOK += nonOK
fmt.Fprintf(&b, "%-20s %8d %8.1f %7s %7s %7s %7.0f %s\n",
op, s.count, float64(s.count)/elapsed,
s.quantile(0.50), s.quantile(0.90), s.quantile(0.99),
float64(s.maxNs)/float64(time.Millisecond), codeBreakdown(s.codes))
}
fmt.Fprintf(&b, "\ntotal calls: %d, throughput: %.1f req/s\n", totalCalls, float64(totalCalls)/elapsed)
rate := 0.0
if totalCalls > 0 {
rate = 100 * float64(totalNonOK) / float64(totalCalls)
}
fmt.Fprintf(&b, "non-ok results: %d (%.2f%%)\n", totalNonOK, rate)
if len(r.events) > 0 {
fmt.Fprintf(&b, "\nlive events:\n")
ekeys := make([]string, 0, len(r.events))
for k := range r.events {
ekeys = append(ekeys, k)
}
sort.Strings(ekeys)
for _, k := range ekeys {
fmt.Fprintf(&b, " %-16s %d\n", k, r.events[k])
}
}
fmt.Fprintf(&b, "stream errors: %d\n", r.streamErrs)
return b.String()
}
// codeBreakdown renders a stat's code counts as "ok:1234 not_your_turn:5 ...",
// highest-count first.
func codeBreakdown(codes map[string]int64) string {
type kv struct {
code string
n int64
}
pairs := make([]kv, 0, len(codes))
for c, n := range codes {
pairs = append(pairs, kv{c, n})
}
sort.Slice(pairs, func(i, j int) bool {
if pairs[i].n != pairs[j].n {
return pairs[i].n > pairs[j].n
}
return pairs[i].code < pairs[j].code
})
parts := make([]string, len(pairs))
for i, p := range pairs {
parts[i] = fmt.Sprintf("%s:%d", p.code, p.n)
}
return strings.Join(parts, " ")
}
+53
View File
@@ -0,0 +1,53 @@
package report
import (
"strings"
"testing"
"time"
)
// TestRecorderTotalsAndSummary checks call/error tallying and that the rendered
// summary surfaces operations, codes, events and stream errors.
func TestRecorderTotalsAndSummary(t *testing.T) {
r := New()
r.Record("game.state", "ok", 5*time.Millisecond)
r.Record("game.state", "ok", 7*time.Millisecond)
r.Record("game.submit_play", "not_your_turn", 3*time.Millisecond)
r.Record("hammer:games.list", "rate_limited", time.Millisecond)
r.Event("your_turn")
r.Event("your_turn")
r.StreamErr()
calls, nonOK := r.Totals()
if calls != 4 {
t.Errorf("calls = %d, want 4", calls)
}
if nonOK != 2 {
t.Errorf("nonOK = %d, want 2", nonOK)
}
s := r.Summary()
for _, want := range []string{"game.state", "not_your_turn", "rate_limited", "your_turn", "stream errors: 1"} {
if !strings.Contains(s, want) {
t.Errorf("summary missing %q\n---\n%s", want, s)
}
}
}
// TestOpStatQuantile checks the bucketed percentile estimate lands on the right bucket
// bound.
func TestOpStatQuantile(t *testing.T) {
s := newOpStat()
for i := 0; i < 90; i++ {
s.record("ok", 10*time.Millisecond)
}
for i := 0; i < 10; i++ {
s.record("ok", 1000*time.Millisecond)
}
if got := s.quantile(0.50); got != "10" {
t.Errorf("p50 = %s, want 10", got)
}
if got := s.quantile(0.99); got != "1000" {
t.Errorf("p99 = %s, want 1000", got)
}
}
+170
View File
@@ -0,0 +1,170 @@
package scenario
import (
"context"
"fmt"
"math/rand"
"time"
"scrabble/loadtest/internal/edge"
"scrabble/loadtest/internal/moves"
"scrabble/loadtest/internal/seed"
)
// Game is one assembled match: its id, variant and members in seat order (Members[0]
// is the inviter, seat 0).
type Game struct {
ID string
Variant string
Members []seed.Account
}
// seatOf returns the seat index of accountID in the game, or -1.
func (g *Game) seatOf(accountID string) int {
for i, m := range g.Members {
if m.ID.String() == accountID {
return i
}
}
return -1
}
// assembleCohort forms games among a cohort of active players via the invitation
// flow, aiming for gamesPerPlayer (3-5) concurrent games per player with 2-4 players
// each. It returns the games it managed to start. Failures are logged and skipped so
// a partial assembly still drives load.
func (d *Driver) assembleCohort(ctx context.Context, cohort []seed.Account, gamesPerPlayer int, rng *rand.Rand) []*Game {
if len(cohort) < 2 {
return nil
}
gamesOf := make(map[string]int, len(cohort))
var games []*Game
for i := range cohort {
inviter := cohort[i]
target := 3 + rng.Intn(3) // 3..5
if gamesPerPlayer > 0 {
target = gamesPerPlayer
}
for gamesOf[inviter.ID.String()] < target {
members := pickMembers(cohort, inviter, rng)
if len(members) < 2 {
break
}
variant := moves.Variants()[rng.Intn(len(moves.Variants()))]
g, err := d.assemble(ctx, members, variant)
if err != nil {
d.log.Debug("assemble game", "err", err)
break
}
games = append(games, g)
for _, m := range members {
gamesOf[m.ID.String()]++
}
}
}
return games
}
// pickMembers builds a 2-4 player group led by inviter, drawing distinct others from
// the cohort at random.
func pickMembers(cohort []seed.Account, inviter seed.Account, rng *rand.Rand) []seed.Account {
size := 2 + rng.Intn(3) // 2..4
members := []seed.Account{inviter}
seen := map[string]bool{inviter.ID.String(): true}
for attempts := 0; len(members) < size && attempts < 4*size; attempts++ {
cand := cohort[rng.Intn(len(cohort))]
if seen[cand.ID.String()] {
continue
}
seen[cand.ID.String()] = true
members = append(members, cand)
}
return members
}
// assemble runs the invitation flow for one game: the inviter (members[0]) invites
// the rest, each invitee accepts the pending invitation, and the completing accept
// starts the game, which is then located in the inviter's game list.
func (d *Driver) assemble(ctx context.Context, members []seed.Account, variant string) (*Game, error) {
inviter := members[0]
inviteeIDs := make([]string, len(members)-1)
for i, m := range members[1:] {
inviteeIDs[i] = m.ID.String()
}
t0 := time.Now()
code, err := d.edge.CreateInvitation(ctx, inviter.Token, inviteeIDs, variant)
d.rec.Record("invitation.create", code, time.Since(t0))
if err != nil || code != "ok" {
return nil, fmt.Errorf("invitation.create: %s", code)
}
for _, invitee := range members[1:] {
t0 = time.Now()
list, lc, err := d.edge.ListInvitations(ctx, invitee.Token)
d.rec.Record("invitation.list", lc, time.Since(t0))
if err != nil || lc != "ok" {
return nil, fmt.Errorf("invitation.list: %s", lc)
}
invID := findPending(list, inviter.ID.String())
if invID == "" {
return nil, fmt.Errorf("no pending invitation from %s", inviter.ID)
}
t0 = time.Now()
ac, err := d.edge.AcceptInvitation(ctx, invitee.Token, invID)
d.rec.Record("invitation.accept", ac, time.Since(t0))
if err != nil || ac != "ok" {
return nil, fmt.Errorf("invitation.accept: %s", ac)
}
}
t0 = time.Now()
games, gc, err := d.edge.GamesList(ctx, inviter.Token)
d.rec.Record("games.list", gc, time.Since(t0))
if err != nil || gc != "ok" {
return nil, fmt.Errorf("games.list: %s", gc)
}
ids := make([]string, len(members))
for i, m := range members {
ids[i] = m.ID.String()
}
gameID := findGame(games, ids)
if gameID == "" {
return nil, fmt.Errorf("started game not found for %d members", len(members))
}
return &Game{ID: gameID, Variant: variant, Members: members}, nil
}
// findPending returns the id of a pending invitation from inviterID, or "".
func findPending(list []edge.Invitation, inviterID string) string {
for _, inv := range list {
if inv.InviterID == inviterID && inv.Status == "pending" {
return inv.ID
}
}
return ""
}
// findGame returns the id of the active game whose seat set equals memberIDs, or "".
func findGame(games []edge.Game, memberIDs []string) string {
want := make(map[string]bool, len(memberIDs))
for _, id := range memberIDs {
want[id] = true
}
for _, g := range games {
if !g.Active() || len(g.Seats) != len(memberIDs) {
continue
}
match := true
for _, s := range g.Seats {
if !want[s] {
match = false
break
}
}
if match {
return g.ID
}
}
return ""
}
+45
View File
@@ -0,0 +1,45 @@
package scenario
import (
"context"
"sync"
"time"
"scrabble/loadtest/internal/seed"
)
// HammerConfig parameterises the gateway-hammer: how many concurrent callers and for
// how long to deliberately exceed the per-user rate limit from a single account.
type HammerConfig struct {
Workers int
Duration time.Duration
}
// DefaultHammer returns a hammer that comfortably exceeds the 300/min per-user limit.
func DefaultHammer() HammerConfig {
return HammerConfig{Workers: 20, Duration: 15 * time.Second}
}
// Hammer drives games.list from a single account far above the per-user rate limit to
// verify the limiter holds — rejections surface as the "rate_limited" code — and to
// measure its cost. Every call is recorded under "hammer:games.list" so the report
// shows the ok/rate_limited split and the rejection latency separately from the
// realistic traffic.
func (d *Driver) Hammer(ctx context.Context, acc seed.Account, cfg HammerConfig) {
runCtx, cancel := context.WithTimeout(ctx, cfg.Duration)
defer cancel()
d.log.Info("gateway-hammer", "workers", cfg.Workers, "duration", cfg.Duration)
var wg sync.WaitGroup
for w := 0; w < cfg.Workers; w++ {
wg.Add(1)
go func() {
defer wg.Done()
for runCtx.Err() == nil {
t0 := time.Now()
_, code, _ := d.edge.GamesList(runCtx, acc.Token)
d.rec.Record("hammer:games.list", code, time.Since(t0))
}
}()
}
wg.Wait()
}
+243
View File
@@ -0,0 +1,243 @@
// Package scenario drives virtual players against the gateway edge protocol: it
// assembles real games through the invitation flow, then runs each player's turn
// loop (poll state, replay history, generate a legal move with the embedded solver,
// submit it) plus a fraction of secondary operations. It exposes the moderate
// realistic ramp agreed for the R2 early pass and a separate gateway-hammer.
package scenario
import (
"context"
"log/slog"
"math/rand"
"sync"
"time"
"scrabble/loadtest/internal/edge"
"scrabble/loadtest/internal/moves"
"scrabble/loadtest/internal/report"
"scrabble/loadtest/internal/seed"
)
// Driver ties the edge client, the local move generator and the run recorder
// together. All three are safe for concurrent use by many player goroutines.
type Driver struct {
edge *edge.Client
moves *moves.Registry
rec *report.Recorder
log *slog.Logger
}
// NewDriver builds a Driver.
func NewDriver(c *edge.Client, m *moves.Registry, rec *report.Recorder, log *slog.Logger) *Driver {
return &Driver{edge: c, moves: m, rec: rec, log: log}
}
// RealisticConfig parameterises the under-the-limit ramp.
type RealisticConfig struct {
Steps []int // concurrent active players per step (cumulative)
StepDur time.Duration // hold time per step
GamesPerPlayer int // target concurrent games per player; 0 => random 3..5
Tick time.Duration // per-player operation cadence (keeps a player under the per-user limit)
SecondaryProb float64 // chance per tick of a non-move operation
}
// DefaultRealistic returns the moderate ramp agreed for the R2 early pass: 50 -> 200
// -> 500 concurrent players, ~12 minutes per step, ~1 op/s per player.
func DefaultRealistic() RealisticConfig {
return RealisticConfig{
Steps: []int{50, 200, 500},
StepDur: 12 * time.Minute,
Tick: 800 * time.Millisecond,
SecondaryProb: 0.08,
}
}
// RunRealistic runs the staged ramp. Each step activates more players (drawn from the
// seeded pool), assembles a cohort of games for them and starts their turn loops; the
// loops run until the whole ramp ends. Players from earlier steps keep playing, so
// load is cumulative.
func (d *Driver) RunRealistic(ctx context.Context, pool *seed.Pool, cfg RealisticConfig) error {
players := shuffledPool(pool)
runCtx, cancel := context.WithCancel(ctx)
defer cancel()
var wg sync.WaitGroup
activated := 0
for si, target := range cfg.Steps {
if target > len(players) {
target = len(players)
}
cohort := players[activated:target]
activated = target
if len(cohort) >= 2 {
rng := rand.New(rand.NewSource(time.Now().UnixNano() + int64(si)))
games := d.assembleCohort(runCtx, cohort, cfg.GamesPerPlayer, rng)
byPlayer := gamesByPlayer(games)
d.log.Info("ramp step", "step", si+1, "active", activated, "cohort", len(cohort), "games", len(games))
for pi := range cohort {
p := cohort[pi]
wg.Add(1)
go func(p seed.Account, pg []*Game, sd int64) {
defer wg.Done()
d.playerLoop(runCtx, p, pg, cfg, rand.New(rand.NewSource(sd)))
}(p, byPlayer[p.ID.String()], time.Now().UnixNano()+int64(pi))
}
} else {
d.log.Warn("ramp step skipped: cohort too small", "step", si+1, "cohort", len(cohort))
}
select {
case <-time.After(cfg.StepDur):
case <-ctx.Done():
cancel()
wg.Wait()
return ctx.Err()
}
}
cancel()
wg.Wait()
return nil
}
// playerLoop runs one virtual player: a live-event subscription (loads the push hub,
// counts events) plus a round-robin turn loop over the player's games.
func (d *Driver) playerLoop(ctx context.Context, p seed.Account, games []*Game, cfg RealisticConfig, rng *rand.Rand) {
go d.subscribeLoop(ctx, p)
if len(games) == 0 {
<-ctx.Done()
return
}
ticker := time.NewTicker(cfg.Tick)
defer ticker.Stop()
gi := 0
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
g := games[gi%len(games)]
gi++
if rng.Float64() < cfg.SecondaryProb {
d.secondaryOp(ctx, p, g, rng)
continue
}
d.playTurn(ctx, p, g, rng)
}
}
}
// subscribeLoop holds the player's live-event stream open, counting events and
// reconnecting with a brief backoff after a drop, until the run ends.
func (d *Driver) subscribeLoop(ctx context.Context, p seed.Account) {
for ctx.Err() == nil {
err := d.edge.Subscribe(ctx, p.Token, func(e edge.Event) { d.rec.Event(e.Kind) })
if ctx.Err() != nil {
return
}
if err != nil {
d.rec.StreamErr()
}
select {
case <-ctx.Done():
return
case <-time.After(time.Second):
}
}
}
// playTurn plays one turn in g when it is the player's move: fetch state, replay
// history, pick a legal move and submit it (or exchange / pass).
func (d *Driver) playTurn(ctx context.Context, p seed.Account, g *Game, rng *rand.Rand) {
seat := g.seatOf(p.ID.String())
if seat < 0 {
return
}
t0 := time.Now()
st, code, err := d.edge.State(ctx, p.Token, g.ID)
d.rec.Record("game.state", code, time.Since(t0))
if err != nil || code != "ok" || !st.Game.Active() || st.Game.ToMove != seat {
return
}
t0 = time.Now()
hist, hc, err := d.edge.History(ctx, p.Token, g.ID)
d.rec.Record("game.history", hc, time.Since(t0))
if err != nil || hc != "ok" {
return
}
action, err := d.moves.Pick(g.Variant, hist, st.Rack, st.BagLen, rng)
if err != nil {
d.log.Debug("pick move", "variant", g.Variant, "err", err)
return
}
switch action.Kind {
case "play":
t0 = time.Now()
_, c, _ := d.edge.SubmitPlay(ctx, p.Token, g.ID, action.Dir, action.Tiles)
d.rec.Record("game.submit_play", c, time.Since(t0))
case "exchange":
t0 = time.Now()
_, c, _ := d.edge.Exchange(ctx, p.Token, g.ID, action.Exchange)
d.rec.Record("game.exchange", c, time.Since(t0))
default:
t0 = time.Now()
_, c, _ := d.edge.Pass(ctx, p.Token, g.ID)
d.rec.Record("game.pass", c, time.Since(t0))
}
}
// secondaryOp exercises one of the non-move edge operations the plan calls out, so
// the run touches nudge / chat / check-word / draft / profile / stats too.
func (d *Driver) secondaryOp(ctx context.Context, p seed.Account, g *Game, rng *rand.Rand) {
t0 := time.Now()
switch rng.Intn(7) {
case 0:
c, _ := d.edge.Nudge(ctx, p.Token, g.ID)
d.rec.Record("chat.nudge", c, time.Since(t0))
case 1:
c, _ := d.edge.ChatPost(ctx, p.Token, g.ID, "gg")
d.rec.Record("chat.post", c, time.Since(t0))
case 2:
c, _ := d.edge.CheckWord(ctx, p.Token, g.ID, []byte{0, 1, 2})
d.rec.Record("game.check_word", c, time.Since(t0))
case 3:
// rack_order is an opaque string and board_tiles a (here empty) array, per the
// backend draft DTO; a malformed shape is rejected as bad_request.
c, _ := d.edge.DraftSave(ctx, p.Token, g.ID, `{"rack_order":"","board_tiles":[]}`)
d.rec.Record("draft.save", c, time.Since(t0))
case 4:
c, _ := d.edge.DraftGet(ctx, p.Token, g.ID)
d.rec.Record("draft.get", c, time.Since(t0))
case 5:
lang := "en"
if rng.Intn(2) == 1 {
lang = "ru"
}
c, _ := d.edge.ProfileUpdate(ctx, p.Token, p.Name, lang)
d.rec.Record("profile.update", c, time.Since(t0))
default:
c, _ := d.edge.Stats(ctx, p.Token)
d.rec.Record("stats.get", c, time.Since(t0))
}
}
// shuffledPool returns every seeded account in random order, so an active set is a
// representative mix of durable and guest accounts.
func shuffledPool(pool *seed.Pool) []seed.Account {
all := pool.All()
rng := rand.New(rand.NewSource(time.Now().UnixNano()))
rng.Shuffle(len(all), func(i, j int) { all[i], all[j] = all[j], all[i] })
return all
}
// gamesByPlayer indexes the assembled games by each member's account id.
func gamesByPlayer(games []*Game) map[string][]*Game {
m := make(map[string][]*Game)
for _, g := range games {
for _, mem := range g.Members {
id := mem.ID.String()
m[id] = append(m[id], g)
}
}
return m
}
+181
View File
@@ -0,0 +1,181 @@
package seed
import (
"context"
"fmt"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// Marker prefixes every display_name the harness writes. Cleanup matches on it, so
// the harness only ever deletes its own rows and never touches real accounts. It is
// a distinctive, letters-only string so a profile.update can resend the seeded name
// through the editable-display-name validator (which forbids digits and colons).
const Marker = "Zzloadtest"
// Schema-qualified targets so the seeder does not depend on the connection's
// search_path (the backend pins search_path=backend; we qualify explicitly).
var (
accountsTbl = pgx.Identifier{"backend", "accounts"}
identitiesTbl = pgx.Identifier{"backend", "identities"}
sessionsTbl = pgx.Identifier{"backend", "sessions"}
)
// Account is one seeded player: its account id, marker display name and the
// plaintext bearer token the driver presents in the Authorization header. Guest
// marks a guest (no identity, accrues no statistics). Name is retained so a
// profile.update can resend the marker display name and keep the row findable by
// Cleanup.
type Account struct {
ID uuid.UUID
Name string
Token string
Guest bool
}
// Pool is the seeded population, split by durability.
type Pool struct {
Guests []Account
Durables []Account
}
// All returns every seeded account, durables first.
func (p *Pool) All() []Account {
out := make([]Account, 0, len(p.Durables)+len(p.Guests))
out = append(out, p.Durables...)
out = append(out, p.Guests...)
return out
}
// Seeder writes and removes the harness population over a pgx pool against the
// backend Postgres schema.
type Seeder struct{ pool *pgxpool.Pool }
// New connects to dsn (the backend Postgres) and verifies the connection.
func New(ctx context.Context, dsn string) (*Seeder, error) {
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
return nil, fmt.Errorf("seed: connect: %w", err)
}
if err := pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("seed: ping: %w", err)
}
return &Seeder{pool: pool}, nil
}
// Close releases the pool.
func (s *Seeder) Close() { s.pool.Close() }
// Seed inserts nDurable durable accounts (each with a confirmed email identity) and
// nGuest guest accounts, an active session per account, and returns the population
// with the plaintext tokens. Rows go in over COPY in foreign-key order (accounts,
// then identities and sessions). Every row carries Marker in its display name /
// external id so Cleanup can find them.
func (s *Seeder) Seed(ctx context.Context, nDurable, nGuest int) (*Pool, error) {
pool := &Pool{
Durables: make([]Account, 0, nDurable),
Guests: make([]Account, 0, nGuest),
}
var acctRows, identRows, sessRows [][]any
add := func(guest bool, i int) error {
aid, err := uuid.NewV7()
if err != nil {
return err
}
sid, err := uuid.NewV7()
if err != nil {
return err
}
token, hash, err := GenerateToken()
if err != nil {
return err
}
lang := "en"
if i%2 == 1 {
lang = "ru"
}
kind := "d"
if guest {
kind = "g"
}
// A letters-only display name (Marker + kind), valid per the editable-name
// validator; account_id, not the name, is the unique key, so duplicates are fine.
name := Marker + kind
acctRows = append(acctRows, []any{aid, name, guest, lang})
sessRows = append(sessRows, []any{sid, aid, hash, "active"})
if !guest {
iid, err := uuid.NewV7()
if err != nil {
return err
}
ext := fmt.Sprintf("%s%s@loadtest.invalid", Marker, aid)
identRows = append(identRows, []any{iid, aid, "email", ext, true})
}
acc := Account{ID: aid, Name: name, Token: token, Guest: guest}
if guest {
pool.Guests = append(pool.Guests, acc)
} else {
pool.Durables = append(pool.Durables, acc)
}
return nil
}
for i := 0; i < nDurable; i++ {
if err := add(false, i); err != nil {
return nil, err
}
}
for i := 0; i < nGuest; i++ {
if err := add(true, i); err != nil {
return nil, err
}
}
if _, err := s.pool.CopyFrom(ctx, accountsTbl,
[]string{"account_id", "display_name", "is_guest", "preferred_language"},
pgx.CopyFromRows(acctRows)); err != nil {
return nil, fmt.Errorf("seed: copy accounts: %w", err)
}
if len(identRows) > 0 {
if _, err := s.pool.CopyFrom(ctx, identitiesTbl,
[]string{"identity_id", "account_id", "kind", "external_id", "confirmed"},
pgx.CopyFromRows(identRows)); err != nil {
return nil, fmt.Errorf("seed: copy identities: %w", err)
}
}
if _, err := s.pool.CopyFrom(ctx, sessionsTbl,
[]string{"session_id", "account_id", "token_hash", "status"},
pgx.CopyFromRows(sessRows)); err != nil {
return nil, fmt.Errorf("seed: copy sessions: %w", err)
}
return pool, nil
}
// Cleanup removes everything the harness created: first the games any harness
// account is seated in (cascading game_players / game_moves / complaints / chat),
// then the harness accounts (cascading identities, sessions, stats, invitations,
// drafts and the rest). It is scoped by Marker, so it is safe to run against a
// contour that also holds real data. The authoritative hard reset remains the
// contour DB wipe (DROP SCHEMA backend CASCADE + backend restart). It returns the
// number of accounts removed.
func (s *Seeder) Cleanup(ctx context.Context) (int, error) {
if _, err := s.pool.Exec(ctx, `
DELETE FROM backend.games
WHERE game_id IN (
SELECT p.game_id FROM backend.game_players p
JOIN backend.accounts a ON a.account_id = p.account_id
WHERE a.display_name LIKE $1
)`, Marker+"%"); err != nil {
return 0, fmt.Errorf("seed: cleanup games: %w", err)
}
tag, err := s.pool.Exec(ctx,
`DELETE FROM backend.accounts WHERE display_name LIKE $1`, Marker+"%")
if err != nil {
return 0, fmt.Errorf("seed: cleanup accounts: %w", err)
}
return int(tag.RowsAffected()), nil
}
+33
View File
@@ -0,0 +1,33 @@
// Package seed creates accounts, identities and sessions directly in the backend
// Postgres schema so the load driver can authenticate as many pre-provisioned
// players without paying the per-IP cost of the auth edge operations. It owns the
// inverse operation too (cleanup of everything it created).
package seed
import (
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
)
// GenerateToken mints an opaque bearer token and its stored hash. token is the
// plaintext handed to the client; hash is what the seeder writes to
// sessions.token_hash. The transformation matches backend/internal/session so a
// resolve of token recomputes the same hash and finds the seeded row.
func GenerateToken() (token, hash string, err error) {
buf := make([]byte, 32)
if _, err := rand.Read(buf); err != nil {
return "", "", err
}
token = base64.RawURLEncoding.EncodeToString(buf)
return token, HashToken(token), nil
}
// HashToken returns the hex-encoded SHA-256 of token. It is the exact hash the
// backend session resolver computes (backend/internal/session/token.go), kept in
// lockstep so seeded sessions validate.
func HashToken(token string) string {
sum := sha256.Sum256([]byte(token))
return hex.EncodeToString(sum[:])
}
+44
View File
@@ -0,0 +1,44 @@
package seed
import (
"crypto/sha256"
"encoding/hex"
"testing"
)
// TestHashTokenMatchesSHA256Hex pins HashToken to the exact transformation the
// backend session resolver uses (hex-encoded SHA-256), the invariant that makes a
// seeded session resolve.
func TestHashTokenMatchesSHA256Hex(t *testing.T) {
const token = "an-example-bearer-token"
sum := sha256.Sum256([]byte(token))
want := hex.EncodeToString(sum[:])
if got := HashToken(token); got != want {
t.Fatalf("HashToken(%q) = %s, want %s", token, got, want)
}
}
// TestGenerateTokenRoundTrip checks that a minted token hashes to the stored hash and
// that tokens are unique.
func TestGenerateTokenRoundTrip(t *testing.T) {
token, hash, err := GenerateToken()
if err != nil {
t.Fatalf("GenerateToken: %v", err)
}
if token == "" || hash == "" {
t.Fatal("empty token or hash")
}
if len(hash) != 64 {
t.Fatalf("hash length = %d, want 64 hex chars", len(hash))
}
if got := HashToken(token); got != hash {
t.Fatalf("hash mismatch: GenerateToken returned %s, HashToken(token) = %s", hash, got)
}
token2, _, err := GenerateToken()
if err != nil {
t.Fatalf("GenerateToken (2nd): %v", err)
}
if token2 == token {
t.Fatal("two generated tokens are identical")
}
}
+4 -3
View File
@@ -1,8 +1,9 @@
# Telegram connector image.
#
# The connector imports only the shared scrabble/pkg module, so the build drops the
# other workspace modules (backend, gateway) and the scrabble-solver replace from a
# copy of go.work: it needs neither their sources nor the solver sibling checkout.
# other workspace modules (backend, gateway, loadtest), the loadtest-only
# scrabble/gateway replace and the scrabble-solver replace from a copy of go.work: it
# needs neither their sources nor the solver sibling checkout.
# Build from the repository ROOT so go.work, pkg/ and platform/telegram/ are all in
# the context (see deploy/docker-compose.yml, which sets context: ../../..).
FROM golang:1.26.3-alpine AS build
@@ -13,7 +14,7 @@ COPY pkg ./pkg
COPY platform/telegram ./platform/telegram
# Reduce the workspace to what the connector needs: only pkg + platform/telegram.
RUN go work edit -dropuse=./backend -dropuse=./gateway -dropreplace=scrabble-solver
RUN go work edit -dropuse=./backend -dropuse=./gateway -dropuse=./loadtest -dropreplace=scrabble/gateway@v0.0.0 -dropreplace=scrabble-solver
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/telegram ./platform/telegram/cmd/telegram