Merge pull request 'fix(edge): suppress dead HTTP/3 advert with Alt-Svc: clear' (#119) from feature/edge-suppress-http3-altsvc into development

deploy/caddy/Caddyfile

@@ -21,6 +21,18 @@
 }
 
 {$CADDY_SITE_ADDRESS::80} {
+	# HTTP/3 is advertised by default whenever this caddy terminates TLS (prod:
+	# CADDY_SITE_ADDRESS is the domain). But UDP/443 is never reachable — the prod
+	# compose maps only "443:443" (TCP) and ufw opens 443/tcp — so a client that cached
+	# the `Alt-Svc: h3` advert (sticky for ma=2592000s) stalls on the dead QUIC path
+	# before falling back to h2, which surfaced as the Telegram Mini App intermittently
+	# hanging on load. `Alt-Svc: clear` actively drops any cached alternative and pins
+	# clients to h2/h1; it is applied site-wide so every route is covered. In the test
+	# contour this caddy serves plain :80 (no h3 to advertise) and the host caddy
+	# re-stamps its own Alt-Svc, so the live test fix lives in the host caddy — here it
+	# is the prod fix. Background + alternatives (incl. serving h3 for real): docs/EDGE_HTTP3.md.
+	header Alt-Svc clear
+
 	# Operator surfaces under /_gm: a single shared Basic-Auth, then route.
 	@gm path /_gm /_gm/*
 	handle @gm {

docs/ARCHITECTURE.md

@@ -1098,7 +1098,10 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
   the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`),
   the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API
   egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS —
-  `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes**
+  `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. Caddy advertises HTTP/3 by default, but UDP/443 is not exposed (the
+  compose maps only TCP and ufw opens 443/tcp), so the edge emits `Alt-Svc: clear` to keep
+  clients on h2/h1 rather than stall on a dead QUIC path — see [`EDGE_HTTP3.md`](EDGE_HTTP3.md).
+  The gateway **publishes**
   the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`,
   ServerName `gateway`, so TLS validation is independent of the public dial address), holds
   no inbound port, and login is unaffected if that host or the link is down.

docs/EDGE_HTTP3.md

@@ -0,0 +1,111 @@
+# Edge HTTP/3 (`Alt-Svc`) policy
+
+## TL;DR
+
+The edge **advertises HTTP/3 but does not actually serve it** (UDP/443 is not exposed),
+so we suppress the advert with `Alt-Svc: clear`. Advertising QUIC on `:443/udp` while
+that port is unreachable makes clients — notably the Telegram Mini App webview — stall
+on a dead QUIC connection before falling back to h2, which shows up as the app "hanging
+on load".
+
+## Symptom
+
+Opening the Mini App intermittently hangs on load: from a barely-noticeable pause to
+several seconds, sometimes a blank window that never finishes downloading `index.html`.
+Intermittent, worse after the first successful visit, reproduced on both the test
+contour and prod.
+
+## Root cause
+
+Caddy enables HTTP/3 by default on any TLS listener and emits
+`Alt-Svc: h3=":443"; ma=2592000` — telling every client "reach me over QUIC/UDP 443"
+and to cache that for 30 days. But UDP/443 is **never reachable end to end**:
+
+- **Test contour**: the host caddy publishes only `:443/tcp` (`docker port caddy` shows
+  no `udp`); QUIC packets from the internet are dropped.
+- **Prod**: `deploy/docker-compose.prod.yml` maps `"443:443"` (Docker = **TCP only**)
+  and `deploy/ansible/roles/main/tasks/main.yml` opens 443 `proto: tcp`. UDP/443 is
+  dropped at both the publish and the firewall.
+
+Caddy *does* bind `udp/443` inside the container and h3 works container-to-container
+(verified `http=3 code=200`), so the listener is healthy — it is simply not exposed.
+
+A client that cached the advert tries QUIC first on later opens, gets no response, and
+waits for the QUIC attempt to time out before falling back to TCP/h2. That wait is the
+stall. The very first visit (no cached `Alt-Svc`) uses h2 and is fast.
+
+The h2/TCP serving path itself is healthy: 30 fresh-TLS requests through the full path
+(host caddy -> contour caddy -> gateway) measured TTFB ~9.5 ms, total ~9.8 ms, no tail;
+`index.html` is ~1 KB.
+
+## Fix in place (option A — suppress the advert)
+
+Emit `Alt-Svc: clear`, which actively drops any cached alternative (better than merely
+deleting the header, which leaves the sticky 30-day cache in place):
+
+- **Prod / repo**: `deploy/caddy/Caddyfile` — a site-level `header Alt-Svc clear` (this
+  caddy terminates TLS in prod).
+- **Test contour**: the host caddy terminates TLS, so the fix lives there (homelab
+  config, outside this repo): `header Alt-Svc clear` on the `scrabble.*` site. The
+  in-compose caddy serves plain `:80` in test and never advertises h3, so the repo
+  directive is a harmless no-op there (the host caddy re-stamps the header).
+
+`header Alt-Svc clear` overrides Caddy's auto-advert (verified) and is site-scoped.
+
+### Verify
+
+The runner/prod host shell cannot reach the Docker bridge IPs directly, so probe from a
+container on the relevant network, using `--resolve` to hit the TLS-terminating caddy by
+its bridge IP (this also bypasses the public-IP NAT hairpin):
+
+```sh
+# <edge-ip> = the TLS-terminating caddy's IP on its network (docker inspect ... )
+docker run --rm --network edge curlimages/curl:latest -sS -D - -o /dev/null \
+  --resolve <host>:443:<edge-ip> https://<host>/telegram/ | grep -iE '^HTTP|^alt-svc'
+# expect: HTTP/2 200, and NO `alt-svc: h3=...` (the header is absent or `alt-svc: clear`)
+```
+
+## If it recurs — alternatives to try
+
+So we do not re-derive the diagnosis from scratch:
+
+1. **Re-confirm the advert is actually suppressed** with the verify command above. A
+   redeploy or a Caddy upgrade could regress it, or a client may still hold a cached
+   `h3` entry that has not yet been replaced by a `clear` (it needs one successful h2
+   response to receive the `clear`).
+2. **Option B — serve HTTP/3 for real** instead of suppressing it. Worth it only if we
+   actually want QUIC (the benefit is marginal for a ~1 KB shell plus hash-immutable
+   cached assets, and it adds UDP/QUIC attack surface):
+   - Publish UDP: add `"443:443/udp"` next to the TCP map in
+     `deploy/docker-compose.prod.yml` (and publish udp/443 on the test host caddy too).
+   - Open the firewall: add a `443 proto: udp` rule in
+     `deploy/ansible/roles/main/tasks/main.yml`.
+   - Drop the `header Alt-Svc clear` so Caddy advertises h3 again.
+   - Verify with an h3 client from inside the network:
+     `docker run --rm --network edge ymuski/curl-http3 curl --http3-only ...` should
+     return `http=3 code=200`.
+3. **Look past the edge** if the advert is suppressed and stalls persist. The h2 path is
+   fast server-side, so a remaining stall is most likely the client network / RTT / the
+   provider, not our stack. Re-run the timing loop (below) to confirm the server is
+   still <~10 ms TTFB before chasing the client side.
+
+## How this was diagnosed (method, to repeat)
+
+- The runner/prod host shell cannot reach the Docker bridge subnets, so all probing runs
+  from a throwaway container on the target network (`docker run --network <net>
+  curlimages/curl`), using `--resolve <host>:443:<edge-ip>` to bypass the public-IP NAT
+  hairpin and exercise the real TLS path.
+- Compare a fresh-connection timing loop (worst case, full TLS each time) against a
+  keepalive batch to separate handshake cost from serving cost:
+
+  ```sh
+  docker run --rm --network edge curlimages/curl:latest sh -c '
+    for i in $(seq 1 30); do
+      curl -sS -o /dev/null --resolve <host>:443:<edge-ip> \
+        -w "http=%{http_version} code=%{http_code} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
+        https://<host>/telegram/
+    done'
+  ```
+
+- `docker port <caddy>` shows whether `udp/443` is actually published; the response
+  `Alt-Svc` header shows what the edge advertises. The two disagreeing is the bug.