scrabble-game/docs/EDGE_HTTP3.md

Edge HTTP/3 (Alt-Svc) policy

TL;DR

The edge advertises HTTP/3 but does not actually serve it (UDP/443 is not exposed), so we suppress the advert with Alt-Svc: clear. Advertising QUIC on :443/udp while that port is unreachable makes clients — notably the Telegram Mini App webview — stall on a dead QUIC connection before falling back to h2, which shows up as the app "hanging on load".

Symptom

Opening the Mini App intermittently hangs on load: from a barely-noticeable pause to several seconds, sometimes a blank window that never finishes downloading index.html. Intermittent, worse after the first successful visit, reproduced on both the test contour and prod.

Root cause

Caddy enables HTTP/3 by default on any TLS listener and emits Alt-Svc: h3=":443"; ma=2592000 — telling every client "reach me over QUIC/UDP 443" and to cache that for 30 days. But UDP/443 is never reachable end to end:

• Test contour: the host caddy publishes only :443/tcp (docker port caddy shows no udp); QUIC packets from the internet are dropped.
• Prod: deploy/docker-compose.prod.yml maps "443:443" (Docker = TCP only) and deploy/ansible/roles/main/tasks/main.yml opens 443 proto: tcp. UDP/443 is dropped at both the publish and the firewall.

Caddy does bind udp/443 inside the container and h3 works container-to-container (verified http=3 code=200), so the listener is healthy — it is simply not exposed.

A client that cached the advert tries QUIC first on later opens, gets no response, and waits for the QUIC attempt to time out before falling back to TCP/h2. That wait is the stall. The very first visit (no cached Alt-Svc) uses h2 and is fast.

The h2/TCP serving path itself is healthy: 30 fresh-TLS requests through the full path (host caddy -> contour caddy -> gateway) measured TTFB ~9.5 ms, total ~9.8 ms, no tail; index.html is ~1 KB.

Fix in place (option A — suppress the advert)

Emit Alt-Svc: clear, which actively drops any cached alternative (better than merely deleting the header, which leaves the sticky 30-day cache in place):

• Prod / repo: deploy/caddy/Caddyfile — a site-level header Alt-Svc clear (this caddy terminates TLS in prod).
• Test contour: the host caddy terminates TLS, so the fix lives there (homelab config, outside this repo): header Alt-Svc clear on the scrabble.* site. The in-compose caddy serves plain :80 in test and never advertises h3, so the repo directive is a harmless no-op there (the host caddy re-stamps the header).

header Alt-Svc clear overrides Caddy's auto-advert (verified) and is site-scoped.

Verify

The runner/prod host shell cannot reach the Docker bridge IPs directly, so probe from a container on the relevant network, using --resolve to hit the TLS-terminating caddy by its bridge IP (this also bypasses the public-IP NAT hairpin):

# <edge-ip> = the TLS-terminating caddy's IP on its network (docker inspect ... )
docker run --rm --network edge curlimages/curl:latest -sS -D - -o /dev/null \
  --resolve <host>:443:<edge-ip> https://<host>/telegram/ | grep -iE '^HTTP|^alt-svc'
# expect: HTTP/2 200, and NO `alt-svc: h3=...` (the header is absent or `alt-svc: clear`)

If it recurs — alternatives to try

So we do not re-derive the diagnosis from scratch:

1. Re-confirm the advert is actually suppressed with the verify command above. A redeploy or a Caddy upgrade could regress it, or a client may still hold a cached h3 entry that has not yet been replaced by a clear (it needs one successful h2 response to receive the clear).
2. Option B — serve HTTP/3 for real instead of suppressing it. Worth it only if we actually want QUIC (the benefit is marginal for a ~1 KB shell plus hash-immutable cached assets, and it adds UDP/QUIC attack surface):
   • Publish UDP: add "443:443/udp" next to the TCP map in deploy/docker-compose.prod.yml (and publish udp/443 on the test host caddy too).
   • Open the firewall: add a 443 proto: udp rule in deploy/ansible/roles/main/tasks/main.yml.
   • Drop the header Alt-Svc clear so Caddy advertises h3 again.
   • Verify with an h3 client from inside the network: docker run --rm --network edge ymuski/curl-http3 curl --http3-only ... should return http=3 code=200.
3. Look past the edge if the advert is suppressed and stalls persist. The h2 path is fast server-side, so a remaining stall is most likely the client network / RTT / the provider, not our stack. Re-run the timing loop (below) to confirm the server is still <~10 ms TTFB before chasing the client side.

How this was diagnosed (method, to repeat)

• The runner/prod host shell cannot reach the Docker bridge subnets, so all probing runs from a throwaway container on the target network (docker run --network <net> curlimages/curl), using --resolve <host>:443:<edge-ip> to bypass the public-IP NAT hairpin and exercise the real TLS path.

• Compare a fresh-connection timing loop (worst case, full TLS each time) against a keepalive batch to separate handshake cost from serving cost:

  docker run --rm --network edge curlimages/curl:latest sh -c '
    for i in $(seq 1 30); do
      curl -sS -o /dev/null --resolve <host>:443:<edge-ip> \
        -w "http=%{http_version} code=%{http_code} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
        https://<host>/telegram/
    done'
  

• docker port <caddy> shows whether udp/443 is actually published; the response Alt-Svc header shows what the edge advertises. The two disagreeing is the bug.