diff --git a/deploy/caddy/Caddyfile b/deploy/caddy/Caddyfile index 051940a..bb28b63 100644 --- a/deploy/caddy/Caddyfile +++ b/deploy/caddy/Caddyfile @@ -21,6 +21,18 @@ } {$CADDY_SITE_ADDRESS::80} { + # HTTP/3 is advertised by default whenever this caddy terminates TLS (prod: + # CADDY_SITE_ADDRESS is the domain). But UDP/443 is never reachable — the prod + # compose maps only "443:443" (TCP) and ufw opens 443/tcp — so a client that cached + # the `Alt-Svc: h3` advert (sticky for ma=2592000s) stalls on the dead QUIC path + # before falling back to h2, which surfaced as the Telegram Mini App intermittently + # hanging on load. `Alt-Svc: clear` actively drops any cached alternative and pins + # clients to h2/h1; it is applied site-wide so every route is covered. In the test + # contour this caddy serves plain :80 (no h3 to advertise) and the host caddy + # re-stamps its own Alt-Svc, so the live test fix lives in the host caddy — here it + # is the prod fix. Background + alternatives (incl. serving h3 for real): docs/EDGE_HTTP3.md. + header Alt-Svc clear + # Operator surfaces under /_gm: a single shared Basic-Auth, then route. @gm path /_gm /_gm/* handle @gm { diff --git a/deploy/grafana/dashboards/resources.json b/deploy/grafana/dashboards/resources.json index a1d503e..663298d 100644 --- a/deploy/grafana/dashboards/resources.json +++ b/deploy/grafana/dashboards/resources.json @@ -36,7 +36,21 @@ "type": "stat", "title": "Database size", "gridPos": { "h": 5, "w": 6, "x": 18, "y": 0 }, - "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] }, + "fieldConfig": { + "defaults": { + "unit": "bytes", + "color": { "mode": "thresholds" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "yellow", "value": 8589934592 }, + { "color": "red", "value": 17179869184 } + ] + } + }, + "overrides": [] + }, "datasource": { "type": "prometheus", "uid": "prometheus" }, "targets": [{ "refId": "A", "expr": "max(pg_database_size_bytes{datname=\"scrabble\"})" }] }, diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index d7d1b29..fa5361f 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1098,7 +1098,10 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`): the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`), the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS — - `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes** + `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. Caddy advertises HTTP/3 by default, but UDP/443 is not exposed (the + compose maps only TCP and ufw opens 443/tcp), so the edge emits `Alt-Svc: clear` to keep + clients on h2/h1 rather than stall on a dead QUIC path — see [`EDGE_HTTP3.md`](EDGE_HTTP3.md). + The gateway **publishes** the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`, ServerName `gateway`, so TLS validation is independent of the public dial address), holds no inbound port, and login is unaffected if that host or the link is down. diff --git a/docs/EDGE_HTTP3.md b/docs/EDGE_HTTP3.md new file mode 100644 index 0000000..85f71e5 --- /dev/null +++ b/docs/EDGE_HTTP3.md @@ -0,0 +1,111 @@ +# Edge HTTP/3 (`Alt-Svc`) policy + +## TL;DR + +The edge **advertises HTTP/3 but does not actually serve it** (UDP/443 is not exposed), +so we suppress the advert with `Alt-Svc: clear`. Advertising QUIC on `:443/udp` while +that port is unreachable makes clients — notably the Telegram Mini App webview — stall +on a dead QUIC connection before falling back to h2, which shows up as the app "hanging +on load". + +## Symptom + +Opening the Mini App intermittently hangs on load: from a barely-noticeable pause to +several seconds, sometimes a blank window that never finishes downloading `index.html`. +Intermittent, worse after the first successful visit, reproduced on both the test +contour and prod. + +## Root cause + +Caddy enables HTTP/3 by default on any TLS listener and emits +`Alt-Svc: h3=":443"; ma=2592000` — telling every client "reach me over QUIC/UDP 443" +and to cache that for 30 days. But UDP/443 is **never reachable end to end**: + +- **Test contour**: the host caddy publishes only `:443/tcp` (`docker port caddy` shows + no `udp`); QUIC packets from the internet are dropped. +- **Prod**: `deploy/docker-compose.prod.yml` maps `"443:443"` (Docker = **TCP only**) + and `deploy/ansible/roles/main/tasks/main.yml` opens 443 `proto: tcp`. UDP/443 is + dropped at both the publish and the firewall. + +Caddy *does* bind `udp/443` inside the container and h3 works container-to-container +(verified `http=3 code=200`), so the listener is healthy — it is simply not exposed. + +A client that cached the advert tries QUIC first on later opens, gets no response, and +waits for the QUIC attempt to time out before falling back to TCP/h2. That wait is the +stall. The very first visit (no cached `Alt-Svc`) uses h2 and is fast. + +The h2/TCP serving path itself is healthy: 30 fresh-TLS requests through the full path +(host caddy -> contour caddy -> gateway) measured TTFB ~9.5 ms, total ~9.8 ms, no tail; +`index.html` is ~1 KB. + +## Fix in place (option A — suppress the advert) + +Emit `Alt-Svc: clear`, which actively drops any cached alternative (better than merely +deleting the header, which leaves the sticky 30-day cache in place): + +- **Prod / repo**: `deploy/caddy/Caddyfile` — a site-level `header Alt-Svc clear` (this + caddy terminates TLS in prod). +- **Test contour**: the host caddy terminates TLS, so the fix lives there (homelab + config, outside this repo): `header Alt-Svc clear` on the `scrabble.*` site. The + in-compose caddy serves plain `:80` in test and never advertises h3, so the repo + directive is a harmless no-op there (the host caddy re-stamps the header). + +`header Alt-Svc clear` overrides Caddy's auto-advert (verified) and is site-scoped. + +### Verify + +The runner/prod host shell cannot reach the Docker bridge IPs directly, so probe from a +container on the relevant network, using `--resolve` to hit the TLS-terminating caddy by +its bridge IP (this also bypasses the public-IP NAT hairpin): + +```sh +# = the TLS-terminating caddy's IP on its network (docker inspect ... ) +docker run --rm --network edge curlimages/curl:latest -sS -D - -o /dev/null \ + --resolve :443: https:///telegram/ | grep -iE '^HTTP|^alt-svc' +# expect: HTTP/2 200, and NO `alt-svc: h3=...` (the header is absent or `alt-svc: clear`) +``` + +## If it recurs — alternatives to try + +So we do not re-derive the diagnosis from scratch: + +1. **Re-confirm the advert is actually suppressed** with the verify command above. A + redeploy or a Caddy upgrade could regress it, or a client may still hold a cached + `h3` entry that has not yet been replaced by a `clear` (it needs one successful h2 + response to receive the `clear`). +2. **Option B — serve HTTP/3 for real** instead of suppressing it. Worth it only if we + actually want QUIC (the benefit is marginal for a ~1 KB shell plus hash-immutable + cached assets, and it adds UDP/QUIC attack surface): + - Publish UDP: add `"443:443/udp"` next to the TCP map in + `deploy/docker-compose.prod.yml` (and publish udp/443 on the test host caddy too). + - Open the firewall: add a `443 proto: udp` rule in + `deploy/ansible/roles/main/tasks/main.yml`. + - Drop the `header Alt-Svc clear` so Caddy advertises h3 again. + - Verify with an h3 client from inside the network: + `docker run --rm --network edge ymuski/curl-http3 curl --http3-only ...` should + return `http=3 code=200`. +3. **Look past the edge** if the advert is suppressed and stalls persist. The h2 path is + fast server-side, so a remaining stall is most likely the client network / RTT / the + provider, not our stack. Re-run the timing loop (below) to confirm the server is + still <~10 ms TTFB before chasing the client side. + +## How this was diagnosed (method, to repeat) + +- The runner/prod host shell cannot reach the Docker bridge subnets, so all probing runs + from a throwaway container on the target network (`docker run --network + curlimages/curl`), using `--resolve :443:` to bypass the public-IP NAT + hairpin and exercise the real TLS path. +- Compare a fresh-connection timing loop (worst case, full TLS each time) against a + keepalive batch to separate handshake cost from serving cost: + + ```sh + docker run --rm --network edge curlimages/curl:latest sh -c ' + for i in $(seq 1 30); do + curl -sS -o /dev/null --resolve :443: \ + -w "http=%{http_version} code=%{http_code} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \ + https:///telegram/ + done' + ``` + +- `docker port ` shows whether `udp/443` is actually published; the response + `Alt-Svc` header shows what the edge advertises. The two disagreeing is the bug.