scrabble-game/docs/EDGE_HTTP3.md

# Edge HTTP/3 (`Alt-Svc`) policy

## TL;DR

The edge **advertises HTTP/3 but does not actually serve it** (UDP/443 is not exposed),
so we suppress the advert with `Alt-Svc: clear`. Advertising QUIC on `:443/udp` while
that port is unreachable makes clients — notably the Telegram Mini App webview — stall
on a dead QUIC connection before falling back to h2, which shows up as the app "hanging
on load".

## Symptom

Opening the Mini App intermittently hangs on load: from a barely-noticeable pause to
several seconds, sometimes a blank window that never finishes downloading `index.html`.
Intermittent, worse after the first successful visit, reproduced on both the test
contour and prod.

## Root cause

Caddy enables HTTP/3 by default on any TLS listener and emits
`Alt-Svc: h3=":443"; ma=2592000` — telling every client "reach me over QUIC/UDP 443"
and to cache that for 30 days. But UDP/443 is **never reachable end to end**:

- **Test contour**: the host caddy publishes only `:443/tcp` (`docker port caddy` shows
  no `udp`); QUIC packets from the internet are dropped.
- **Prod**: `deploy/docker-compose.prod.yml` maps `"443:443"` (Docker = **TCP only**)
  and `deploy/ansible/roles/main/tasks/main.yml` opens 443 `proto: tcp`. UDP/443 is
  dropped at both the publish and the firewall.

Caddy *does* bind `udp/443` inside the container and h3 works container-to-container
(verified `http=3 code=200`), so the listener is healthy — it is simply not exposed.

A client that cached the advert tries QUIC first on later opens, gets no response, and
waits for the QUIC attempt to time out before falling back to TCP/h2. That wait is the
stall. The very first visit (no cached `Alt-Svc`) uses h2 and is fast.

The h2/TCP serving path itself is healthy: 30 fresh-TLS requests through the full path
(host caddy -> contour caddy -> gateway) measured TTFB ~9.5 ms, total ~9.8 ms, no tail;
`index.html` is ~1 KB.

## Fix in place (option A — suppress the advert)

Emit `Alt-Svc: clear`, which actively drops any cached alternative (better than merely
deleting the header, which leaves the sticky 30-day cache in place):

- **Prod / repo**: `deploy/caddy/Caddyfile` — a site-level `header Alt-Svc clear` (this
  caddy terminates TLS in prod).
- **Test contour**: the host caddy terminates TLS, so the fix lives there (homelab
  config, outside this repo): `header Alt-Svc clear` on the `scrabble.*` site. The
  in-compose caddy serves plain `:80` in test and never advertises h3, so the repo
  directive is a harmless no-op there (the host caddy re-stamps the header).

`header Alt-Svc clear` overrides Caddy's auto-advert (verified) and is site-scoped.

### Verify

The runner/prod host shell cannot reach the Docker bridge IPs directly, so probe from a
container on the relevant network, using `--resolve` to hit the TLS-terminating caddy by
its bridge IP (this also bypasses the public-IP NAT hairpin):

```sh
# <edge-ip> = the TLS-terminating caddy's IP on its network (docker inspect ... )
docker run --rm --network edge curlimages/curl:latest -sS -D - -o /dev/null \
  --resolve <host>:443:<edge-ip> https://<host>/telegram/ | grep -iE '^HTTP|^alt-svc'
# expect: HTTP/2 200, and NO `alt-svc: h3=...` (the header is absent or `alt-svc: clear`)
```

## If it recurs — alternatives to try

So we do not re-derive the diagnosis from scratch:

1. **Re-confirm the advert is actually suppressed** with the verify command above. A
   redeploy or a Caddy upgrade could regress it, or a client may still hold a cached
   `h3` entry that has not yet been replaced by a `clear` (it needs one successful h2
   response to receive the `clear`).
2. **Option B — serve HTTP/3 for real** instead of suppressing it. Worth it only if we
   actually want QUIC (the benefit is marginal for a ~1 KB shell plus hash-immutable
   cached assets, and it adds UDP/QUIC attack surface):
   - Publish UDP: add `"443:443/udp"` next to the TCP map in
     `deploy/docker-compose.prod.yml` (and publish udp/443 on the test host caddy too).
   - Open the firewall: add a `443 proto: udp` rule in
     `deploy/ansible/roles/main/tasks/main.yml`.
   - Drop the `header Alt-Svc clear` so Caddy advertises h3 again.
   - Verify with an h3 client from inside the network:
     `docker run --rm --network edge ymuski/curl-http3 curl --http3-only ...` should
     return `http=3 code=200`.
3. **Look past the edge** if the advert is suppressed and stalls persist. The h2 path is
   fast server-side, so a remaining stall is most likely the client network / RTT / the
   provider, not our stack. Re-run the timing loop (below) to confirm the server is
   still <~10 ms TTFB before chasing the client side.

## How this was diagnosed (method, to repeat)

- The runner/prod host shell cannot reach the Docker bridge subnets, so all probing runs
  from a throwaway container on the target network (`docker run --network <net>
  curlimages/curl`), using `--resolve <host>:443:<edge-ip>` to bypass the public-IP NAT
  hairpin and exercise the real TLS path.
- Compare a fresh-connection timing loop (worst case, full TLS each time) against a
  keepalive batch to separate handshake cost from serving cost:

  ```sh
  docker run --rm --network edge curlimages/curl:latest sh -c '
    for i in $(seq 1 30); do
      curl -sS -o /dev/null --resolve <host>:443:<edge-ip> \
        -w "http=%{http_version} code=%{http_code} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
        https://<host>/telegram/
    done'
  ```

- `docker port <caddy>` shows whether `udp/443` is actually published; the response
  `Alt-Svc` header shows what the edge advertises. The two disagreeing is the bug.