Merge pull request 'Promote development → master: banner tip set + banner/push language fix' (#114) from development into master

deploy/README.md

@@ -150,9 +150,7 @@ Re-run `ansible/` after a host resize — it is idempotent.
 workflow manually (Gitea → Actions → prod-deploy → run from `master`, input
 `confirm=deploy`). It builds + pushes the images to the registry, ships the
 compose/config/certs/env over SSH, deploys the main host with `prod-deploy.sh` (rolling,
-health-gated, **auto-rollback to the previous tag**; caddy is force-recreated on its roll so
+health-gated, **auto-rollback to the previous tag**), then the bot host, then probes the
-a bind-mounted `Caddyfile` change applies — its image is pinned and admin is off, so neither a
-new tag nor a hot reload would pick it up), then the bot host, then probes the
 public site. After `master` is green this workflow is the **only** thing that touches
 prod — nothing auto-deploys there. It runs four visible jobs: **build → deploy-main →
 deploy-bot → verify** (the per-service rolling shows in the deploy-main log).

deploy/caddy/Caddyfile

@@ -21,18 +21,6 @@
 }
 
 {$CADDY_SITE_ADDRESS::80} {
-	# HTTP/3 is advertised by default whenever this caddy terminates TLS (prod:
-	# CADDY_SITE_ADDRESS is the domain). But UDP/443 is never reachable — the prod
-	# compose maps only "443:443" (TCP) and ufw opens 443/tcp — so a client that cached
-	# the `Alt-Svc: h3` advert (sticky for ma=2592000s) stalls on the dead QUIC path
-	# before falling back to h2, which surfaced as the Telegram Mini App intermittently
-	# hanging on load. `Alt-Svc: clear` actively drops any cached alternative and pins
-	# clients to h2/h1; it is applied site-wide so every route is covered. In the test
-	# contour this caddy serves plain :80 (no h3 to advertise) and the host caddy
-	# re-stamps its own Alt-Svc, so the live test fix lives in the host caddy — here it
-	# is the prod fix. Background + alternatives (incl. serving h3 for real): docs/EDGE_HTTP3.md.
-	header Alt-Svc clear
-
 	# Operator surfaces under /_gm: a single shared Basic-Auth, then route.
 	@gm path /_gm /_gm/*
 	handle @gm {

deploy/grafana/dashboards/resources.json

@@ -36,21 +36,7 @@
       "type": "stat",
       "title": "Database size",
       "gridPos": { "h": 5, "w": 6, "x": 18, "y": 0 },
-      "fieldConfig": {
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
-        "defaults": {
-          "unit": "bytes",
-          "color": { "mode": "thresholds" },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              { "color": "green", "value": null },
-              { "color": "yellow", "value": 8589934592 },
-              { "color": "red", "value": 17179869184 }
-            ]
-          }
-        },
-        "overrides": []
-      },
       "datasource": { "type": "prometheus", "uid": "prometheus" },
       "targets": [{ "refId": "A", "expr": "max(pg_database_size_bytes{datname=\"scrabble\"})" }]
     },

deploy/prod-deploy.sh

@@ -74,13 +74,7 @@ health_running() { # health_running <container>: running, not restarting, stable
 roll() { # roll <service> <health-cmd...>
   local svc="$1"; shift
   echo ">>> rolling $svc -> $TAG"
-  # caddy's image is pinned (caddy:2-alpine, no $TAG) and its Caddyfile is bind-mounted, so a
+  dc up -d --no-build --no-deps "$svc" || return 1
-  # config-only change leaves the compose definition unchanged: `up -d` treats the container as
-  # current and does not recreate it, and admin is off so there is no hot reload — the new
-  # Caddyfile would never load. Force a recreate for caddy so config changes always apply; every
-  # other service already recreates on its new $TAG image.
-  local recreate=(); [ "$svc" = caddy ] && recreate=(--force-recreate)
-  dc up -d --no-build --no-deps "${recreate[@]}" "$svc" || return 1
   "$@" || { echo "!!! $svc failed health check"; return 1; }
   echo "<<< $svc healthy"
 }

docs/ARCHITECTURE.md

@@ -1098,10 +1098,7 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
   the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`),
   the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API
   egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS —
-  `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. Caddy advertises HTTP/3 by default, but UDP/443 is not exposed (the
+  `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes**
-  compose maps only TCP and ufw opens 443/tcp), so the edge emits `Alt-Svc: clear` to keep
-  clients on h2/h1 rather than stall on a dead QUIC path — see [`EDGE_HTTP3.md`](EDGE_HTTP3.md).
-  The gateway **publishes**
   the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`,
   ServerName `gateway`, so TLS validation is independent of the public dial address), holds
   no inbound port, and login is unaffected if that host or the link is down.

docs/EDGE_HTTP3.md

@@ -1,111 +0,0 @@
-# Edge HTTP/3 (`Alt-Svc`) policy
-
-## TL;DR
-
-The edge **advertises HTTP/3 but does not actually serve it** (UDP/443 is not exposed),
-so we suppress the advert with `Alt-Svc: clear`. Advertising QUIC on `:443/udp` while
-that port is unreachable makes clients — notably the Telegram Mini App webview — stall
-on a dead QUIC connection before falling back to h2, which shows up as the app "hanging
-on load".
-
-## Symptom
-
-Opening the Mini App intermittently hangs on load: from a barely-noticeable pause to
-several seconds, sometimes a blank window that never finishes downloading `index.html`.
-Intermittent, worse after the first successful visit, reproduced on both the test
-contour and prod.
-
-## Root cause
-
-Caddy enables HTTP/3 by default on any TLS listener and emits
-`Alt-Svc: h3=":443"; ma=2592000` — telling every client "reach me over QUIC/UDP 443"
-and to cache that for 30 days. But UDP/443 is **never reachable end to end**:
-
-- **Test contour**: the host caddy publishes only `:443/tcp` (`docker port caddy` shows
-  no `udp`); QUIC packets from the internet are dropped.
-- **Prod**: `deploy/docker-compose.prod.yml` maps `"443:443"` (Docker = **TCP only**)
-  and `deploy/ansible/roles/main/tasks/main.yml` opens 443 `proto: tcp`. UDP/443 is
-  dropped at both the publish and the firewall.
-
-Caddy *does* bind `udp/443` inside the container and h3 works container-to-container
-(verified `http=3 code=200`), so the listener is healthy — it is simply not exposed.
-
-A client that cached the advert tries QUIC first on later opens, gets no response, and
-waits for the QUIC attempt to time out before falling back to TCP/h2. That wait is the
-stall. The very first visit (no cached `Alt-Svc`) uses h2 and is fast.
-
-The h2/TCP serving path itself is healthy: 30 fresh-TLS requests through the full path
-(host caddy -> contour caddy -> gateway) measured TTFB ~9.5 ms, total ~9.8 ms, no tail;
-`index.html` is ~1 KB.
-
-## Fix in place (option A — suppress the advert)
-
-Emit `Alt-Svc: clear`, which actively drops any cached alternative (better than merely
-deleting the header, which leaves the sticky 30-day cache in place):
-
-- **Prod / repo**: `deploy/caddy/Caddyfile` — a site-level `header Alt-Svc clear` (this
-  caddy terminates TLS in prod).
-- **Test contour**: the host caddy terminates TLS, so the fix lives there (homelab
-  config, outside this repo): `header Alt-Svc clear` on the `scrabble.*` site. The
-  in-compose caddy serves plain `:80` in test and never advertises h3, so the repo
-  directive is a harmless no-op there (the host caddy re-stamps the header).
-
-`header Alt-Svc clear` overrides Caddy's auto-advert (verified) and is site-scoped.
-
-### Verify
-
-The runner/prod host shell cannot reach the Docker bridge IPs directly, so probe from a
-container on the relevant network, using `--resolve` to hit the TLS-terminating caddy by
-its bridge IP (this also bypasses the public-IP NAT hairpin):
-
-```sh
-# <edge-ip> = the TLS-terminating caddy's IP on its network (docker inspect ... )
-docker run --rm --network edge curlimages/curl:latest -sS -D - -o /dev/null \
-  --resolve <host>:443:<edge-ip> https://<host>/telegram/ | grep -iE '^HTTP|^alt-svc'
-# expect: HTTP/2 200, and NO `alt-svc: h3=...` (the header is absent or `alt-svc: clear`)
-```
-
-## If it recurs — alternatives to try
-
-So we do not re-derive the diagnosis from scratch:
-
-1. **Re-confirm the advert is actually suppressed** with the verify command above. A
-   redeploy or a Caddy upgrade could regress it, or a client may still hold a cached
-   `h3` entry that has not yet been replaced by a `clear` (it needs one successful h2
-   response to receive the `clear`).
-2. **Option B — serve HTTP/3 for real** instead of suppressing it. Worth it only if we
-   actually want QUIC (the benefit is marginal for a ~1 KB shell plus hash-immutable
-   cached assets, and it adds UDP/QUIC attack surface):
-   - Publish UDP: add `"443:443/udp"` next to the TCP map in
-     `deploy/docker-compose.prod.yml` (and publish udp/443 on the test host caddy too).
-   - Open the firewall: add a `443 proto: udp` rule in
-     `deploy/ansible/roles/main/tasks/main.yml`.
-   - Drop the `header Alt-Svc clear` so Caddy advertises h3 again.
-   - Verify with an h3 client from inside the network:
-     `docker run --rm --network edge ymuski/curl-http3 curl --http3-only ...` should
-     return `http=3 code=200`.
-3. **Look past the edge** if the advert is suppressed and stalls persist. The h2 path is
-   fast server-side, so a remaining stall is most likely the client network / RTT / the
-   provider, not our stack. Re-run the timing loop (below) to confirm the server is
-   still <~10 ms TTFB before chasing the client side.
-
-## How this was diagnosed (method, to repeat)
-
-- The runner/prod host shell cannot reach the Docker bridge subnets, so all probing runs
-  from a throwaway container on the target network (`docker run --network <net>
-  curlimages/curl`), using `--resolve <host>:443:<edge-ip>` to bypass the public-IP NAT
-  hairpin and exercise the real TLS path.
-- Compare a fresh-connection timing loop (worst case, full TLS each time) against a
-  keepalive batch to separate handshake cost from serving cost:
-
-  ```sh
-  docker run --rm --network edge curlimages/curl:latest sh -c '
-    for i in $(seq 1 30); do
-      curl -sS -o /dev/null --resolve <host>:443:<edge-ip> \
-        -w "http=%{http_version} code=%{http_code} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
-        https://<host>/telegram/
-    done'
-  ```
-
-- `docker port <caddy>` shows whether `udp/443` is actually published; the response
-  `Alt-Svc` header shows what the edge advertises. The two disagreeing is the bug.