From 9253b1bdca68ea44b659b3f71fa027106d84c646 Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Mon, 22 Jun 2026 21:20:31 +0200 Subject: [PATCH] fix(edge): suppress dead HTTP/3 advert with Alt-Svc: clear Caddy enables HTTP/3 by default on any TLS listener and emits Alt-Svc: h3=":443"; ma=2592000, but UDP/443 is never reachable: the prod compose maps only "443:443" (TCP) and ufw opens 443/tcp (test contour: the host caddy publishes only :443/tcp). A client that cached the 30-day advert tries QUIC first on later opens, gets no response, and waits for the QUIC attempt to time out before falling back to h2 -- which surfaced as the Telegram Mini App intermittently hanging on load (a barely-noticeable pause up to a blank window). The h2/TCP serving path itself is healthy (~10ms TTFB). Emit Alt-Svc: clear site-wide at the contour caddy so clients actively drop any cached alternative and stay on h2/h1. This caddy terminates TLS in prod (the fix target); in the test contour it serves plain :80 and the host caddy re-stamps its own Alt-Svc, so the live test fix lives in the host caddy. Add docs/EDGE_HTTP3.md (symptom, diagnosis method, verify, and option B -- serving h3 for real -- if it recurs) and link it from ARCHITECTURE.md. --- deploy/caddy/Caddyfile | 12 +++++ docs/ARCHITECTURE.md | 5 +- docs/EDGE_HTTP3.md | 111 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 docs/EDGE_HTTP3.md diff --git a/deploy/caddy/Caddyfile b/deploy/caddy/Caddyfile index 051940a..bb28b63 100644 --- a/deploy/caddy/Caddyfile +++ b/deploy/caddy/Caddyfile @@ -21,6 +21,18 @@ } {$CADDY_SITE_ADDRESS::80} { + # HTTP/3 is advertised by default whenever this caddy terminates TLS (prod: + # CADDY_SITE_ADDRESS is the domain). But UDP/443 is never reachable — the prod + # compose maps only "443:443" (TCP) and ufw opens 443/tcp — so a client that cached + # the `Alt-Svc: h3` advert (sticky for ma=2592000s) stalls on the dead QUIC path + # before falling back to h2, which surfaced as the Telegram Mini App intermittently + # hanging on load. `Alt-Svc: clear` actively drops any cached alternative and pins + # clients to h2/h1; it is applied site-wide so every route is covered. In the test + # contour this caddy serves plain :80 (no h3 to advertise) and the host caddy + # re-stamps its own Alt-Svc, so the live test fix lives in the host caddy — here it + # is the prod fix. Background + alternatives (incl. serving h3 for real): docs/EDGE_HTTP3.md. + header Alt-Svc clear + # Operator surfaces under /_gm: a single shared Basic-Auth, then route. @gm path /_gm /_gm/* handle @gm { diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index d7d1b29..fa5361f 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1098,7 +1098,10 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`): the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`), the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS — - `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes** + `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. Caddy advertises HTTP/3 by default, but UDP/443 is not exposed (the + compose maps only TCP and ufw opens 443/tcp), so the edge emits `Alt-Svc: clear` to keep + clients on h2/h1 rather than stall on a dead QUIC path — see [`EDGE_HTTP3.md`](EDGE_HTTP3.md). + The gateway **publishes** the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`, ServerName `gateway`, so TLS validation is independent of the public dial address), holds no inbound port, and login is unaffected if that host or the link is down. diff --git a/docs/EDGE_HTTP3.md b/docs/EDGE_HTTP3.md new file mode 100644 index 0000000..85f71e5 --- /dev/null +++ b/docs/EDGE_HTTP3.md @@ -0,0 +1,111 @@ +# Edge HTTP/3 (`Alt-Svc`) policy + +## TL;DR + +The edge **advertises HTTP/3 but does not actually serve it** (UDP/443 is not exposed), +so we suppress the advert with `Alt-Svc: clear`. Advertising QUIC on `:443/udp` while +that port is unreachable makes clients — notably the Telegram Mini App webview — stall +on a dead QUIC connection before falling back to h2, which shows up as the app "hanging +on load". + +## Symptom + +Opening the Mini App intermittently hangs on load: from a barely-noticeable pause to +several seconds, sometimes a blank window that never finishes downloading `index.html`. +Intermittent, worse after the first successful visit, reproduced on both the test +contour and prod. + +## Root cause + +Caddy enables HTTP/3 by default on any TLS listener and emits +`Alt-Svc: h3=":443"; ma=2592000` — telling every client "reach me over QUIC/UDP 443" +and to cache that for 30 days. But UDP/443 is **never reachable end to end**: + +- **Test contour**: the host caddy publishes only `:443/tcp` (`docker port caddy` shows + no `udp`); QUIC packets from the internet are dropped. +- **Prod**: `deploy/docker-compose.prod.yml` maps `"443:443"` (Docker = **TCP only**) + and `deploy/ansible/roles/main/tasks/main.yml` opens 443 `proto: tcp`. UDP/443 is + dropped at both the publish and the firewall. + +Caddy *does* bind `udp/443` inside the container and h3 works container-to-container +(verified `http=3 code=200`), so the listener is healthy — it is simply not exposed. + +A client that cached the advert tries QUIC first on later opens, gets no response, and +waits for the QUIC attempt to time out before falling back to TCP/h2. That wait is the +stall. The very first visit (no cached `Alt-Svc`) uses h2 and is fast. + +The h2/TCP serving path itself is healthy: 30 fresh-TLS requests through the full path +(host caddy -> contour caddy -> gateway) measured TTFB ~9.5 ms, total ~9.8 ms, no tail; +`index.html` is ~1 KB. + +## Fix in place (option A — suppress the advert) + +Emit `Alt-Svc: clear`, which actively drops any cached alternative (better than merely +deleting the header, which leaves the sticky 30-day cache in place): + +- **Prod / repo**: `deploy/caddy/Caddyfile` — a site-level `header Alt-Svc clear` (this + caddy terminates TLS in prod). +- **Test contour**: the host caddy terminates TLS, so the fix lives there (homelab + config, outside this repo): `header Alt-Svc clear` on the `scrabble.*` site. The + in-compose caddy serves plain `:80` in test and never advertises h3, so the repo + directive is a harmless no-op there (the host caddy re-stamps the header). + +`header Alt-Svc clear` overrides Caddy's auto-advert (verified) and is site-scoped. + +### Verify + +The runner/prod host shell cannot reach the Docker bridge IPs directly, so probe from a +container on the relevant network, using `--resolve` to hit the TLS-terminating caddy by +its bridge IP (this also bypasses the public-IP NAT hairpin): + +```sh +# = the TLS-terminating caddy's IP on its network (docker inspect ... ) +docker run --rm --network edge curlimages/curl:latest -sS -D - -o /dev/null \ + --resolve :443: https:///telegram/ | grep -iE '^HTTP|^alt-svc' +# expect: HTTP/2 200, and NO `alt-svc: h3=...` (the header is absent or `alt-svc: clear`) +``` + +## If it recurs — alternatives to try + +So we do not re-derive the diagnosis from scratch: + +1. **Re-confirm the advert is actually suppressed** with the verify command above. A + redeploy or a Caddy upgrade could regress it, or a client may still hold a cached + `h3` entry that has not yet been replaced by a `clear` (it needs one successful h2 + response to receive the `clear`). +2. **Option B — serve HTTP/3 for real** instead of suppressing it. Worth it only if we + actually want QUIC (the benefit is marginal for a ~1 KB shell plus hash-immutable + cached assets, and it adds UDP/QUIC attack surface): + - Publish UDP: add `"443:443/udp"` next to the TCP map in + `deploy/docker-compose.prod.yml` (and publish udp/443 on the test host caddy too). + - Open the firewall: add a `443 proto: udp` rule in + `deploy/ansible/roles/main/tasks/main.yml`. + - Drop the `header Alt-Svc clear` so Caddy advertises h3 again. + - Verify with an h3 client from inside the network: + `docker run --rm --network edge ymuski/curl-http3 curl --http3-only ...` should + return `http=3 code=200`. +3. **Look past the edge** if the advert is suppressed and stalls persist. The h2 path is + fast server-side, so a remaining stall is most likely the client network / RTT / the + provider, not our stack. Re-run the timing loop (below) to confirm the server is + still <~10 ms TTFB before chasing the client side. + +## How this was diagnosed (method, to repeat) + +- The runner/prod host shell cannot reach the Docker bridge subnets, so all probing runs + from a throwaway container on the target network (`docker run --network + curlimages/curl`), using `--resolve :443:` to bypass the public-IP NAT + hairpin and exercise the real TLS path. +- Compare a fresh-connection timing loop (worst case, full TLS each time) against a + keepalive batch to separate handshake cost from serving cost: + + ```sh + docker run --rm --network edge curlimages/curl:latest sh -c ' + for i in $(seq 1 30); do + curl -sS -o /dev/null --resolve :443: \ + -w "http=%{http_version} code=%{http_code} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \ + https:///telegram/ + done' + ``` + +- `docker port ` shows whether `udp/443` is actually published; the response + `Alt-Svc` header shows what the edge advertises. The two disagreeing is the bug.