Merge pull request 'fix(deploy): force-recreate caddy on its roll so config-only changes apply' (#122) from feature/prod-deploy-force-recreate-caddy into development

The prod rolling deploy rolls each service with `compose up -d --no-deps <svc>`.
For caddy that is a no-op on a config-only release: its image is pinned
(caddy:2-alpine, no $TAG), so the compose definition is unchanged between
releases, compose treats the container as current and does not recreate it, and
admin is off so there is no hot reload. The new bind-mounted Caddyfile is seeded
to the host but never loaded -- the v1.2.2 `Alt-Svc: clear` edge fix deployed
green yet did not take effect until caddy was restarted by hand.

Force a recreate for caddy on its roll (every other service already recreates on
its new $TAG image), so a bind-mounted Caddyfile change always applies. Costs a
~1-2s caddy blip per deploy, acceptable for the infrequent manual prod rollout.

deploy/README.md

@@ -150,7 +150,9 @@ Re-run `ansible/` after a host resize — it is idempotent.
 workflow manually (Gitea → Actions → prod-deploy → run from `master`, input
 `confirm=deploy`). It builds + pushes the images to the registry, ships the
 compose/config/certs/env over SSH, deploys the main host with `prod-deploy.sh` (rolling,
-health-gated, **auto-rollback to the previous tag**), then the bot host, then probes the
+health-gated, **auto-rollback to the previous tag**; caddy is force-recreated on its roll so
+a bind-mounted `Caddyfile` change applies — its image is pinned and admin is off, so neither a
+new tag nor a hot reload would pick it up), then the bot host, then probes the
 public site. After `master` is green this workflow is the **only** thing that touches
 prod — nothing auto-deploys there. It runs four visible jobs: **build → deploy-main →
 deploy-bot → verify** (the per-service rolling shows in the deploy-main log).

deploy/caddy/Caddyfile

@@ -21,6 +21,18 @@
 }
 
 {$CADDY_SITE_ADDRESS::80} {
+	# HTTP/3 is advertised by default whenever this caddy terminates TLS (prod:
+	# CADDY_SITE_ADDRESS is the domain). But UDP/443 is never reachable — the prod
+	# compose maps only "443:443" (TCP) and ufw opens 443/tcp — so a client that cached
+	# the `Alt-Svc: h3` advert (sticky for ma=2592000s) stalls on the dead QUIC path
+	# before falling back to h2, which surfaced as the Telegram Mini App intermittently
+	# hanging on load. `Alt-Svc: clear` actively drops any cached alternative and pins
+	# clients to h2/h1; it is applied site-wide so every route is covered. In the test
+	# contour this caddy serves plain :80 (no h3 to advertise) and the host caddy
+	# re-stamps its own Alt-Svc, so the live test fix lives in the host caddy — here it
+	# is the prod fix. Background + alternatives (incl. serving h3 for real): docs/EDGE_HTTP3.md.
+	header Alt-Svc clear
+
 	# Operator surfaces under /_gm: a single shared Basic-Auth, then route.
 	@gm path /_gm /_gm/*
 	handle @gm {

deploy/grafana/dashboards/resources.json

@@ -36,7 +36,21 @@
       "type": "stat",
       "title": "Database size",
       "gridPos": { "h": 5, "w": 6, "x": 18, "y": 0 },
-      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes",
+          "color": { "mode": "thresholds" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 8589934592 },
+              { "color": "red", "value": 17179869184 }
+            ]
+          }
+        },
+        "overrides": []
+      },
       "datasource": { "type": "prometheus", "uid": "prometheus" },
       "targets": [{ "refId": "A", "expr": "max(pg_database_size_bytes{datname=\"scrabble\"})" }]
     },

deploy/prod-deploy.sh

@@ -74,7 +74,13 @@ health_running() { # health_running <container>: running, not restarting, stable
 roll() { # roll <service> <health-cmd...>
   local svc="$1"; shift
   echo ">>> rolling $svc -> $TAG"
-  dc up -d --no-build --no-deps "$svc" || return 1
+  # caddy's image is pinned (caddy:2-alpine, no $TAG) and its Caddyfile is bind-mounted, so a
+  # config-only change leaves the compose definition unchanged: `up -d` treats the container as
+  # current and does not recreate it, and admin is off so there is no hot reload — the new
+  # Caddyfile would never load. Force a recreate for caddy so config changes always apply; every
+  # other service already recreates on its new $TAG image.
+  local recreate=(); [ "$svc" = caddy ] && recreate=(--force-recreate)
+  dc up -d --no-build --no-deps "${recreate[@]}" "$svc" || return 1
   "$@" || { echo "!!! $svc failed health check"; return 1; }
   echo "<<< $svc healthy"
 }

docs/ARCHITECTURE.md

@@ -1098,7 +1098,10 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
   the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`),
   the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API
   egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS —
-  `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes**
+  `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. Caddy advertises HTTP/3 by default, but UDP/443 is not exposed (the
+  compose maps only TCP and ufw opens 443/tcp), so the edge emits `Alt-Svc: clear` to keep
+  clients on h2/h1 rather than stall on a dead QUIC path — see [`EDGE_HTTP3.md`](EDGE_HTTP3.md).
+  The gateway **publishes**
   the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`,
   ServerName `gateway`, so TLS validation is independent of the public dial address), holds
   no inbound port, and login is unaffected if that host or the link is down.

docs/EDGE_HTTP3.md

@@ -0,0 +1,111 @@
+# Edge HTTP/3 (`Alt-Svc`) policy
+
+## TL;DR
+
+The edge **advertises HTTP/3 but does not actually serve it** (UDP/443 is not exposed),
+so we suppress the advert with `Alt-Svc: clear`. Advertising QUIC on `:443/udp` while
+that port is unreachable makes clients — notably the Telegram Mini App webview — stall
+on a dead QUIC connection before falling back to h2, which shows up as the app "hanging
+on load".
+
+## Symptom
+
+Opening the Mini App intermittently hangs on load: from a barely-noticeable pause to
+several seconds, sometimes a blank window that never finishes downloading `index.html`.
+Intermittent, worse after the first successful visit, reproduced on both the test
+contour and prod.
+
+## Root cause
+
+Caddy enables HTTP/3 by default on any TLS listener and emits
+`Alt-Svc: h3=":443"; ma=2592000` — telling every client "reach me over QUIC/UDP 443"
+and to cache that for 30 days. But UDP/443 is **never reachable end to end**:
+
+- **Test contour**: the host caddy publishes only `:443/tcp` (`docker port caddy` shows
+  no `udp`); QUIC packets from the internet are dropped.
+- **Prod**: `deploy/docker-compose.prod.yml` maps `"443:443"` (Docker = **TCP only**)
+  and `deploy/ansible/roles/main/tasks/main.yml` opens 443 `proto: tcp`. UDP/443 is
+  dropped at both the publish and the firewall.
+
+Caddy *does* bind `udp/443` inside the container and h3 works container-to-container
+(verified `http=3 code=200`), so the listener is healthy — it is simply not exposed.
+
+A client that cached the advert tries QUIC first on later opens, gets no response, and
+waits for the QUIC attempt to time out before falling back to TCP/h2. That wait is the
+stall. The very first visit (no cached `Alt-Svc`) uses h2 and is fast.
+
+The h2/TCP serving path itself is healthy: 30 fresh-TLS requests through the full path
+(host caddy -> contour caddy -> gateway) measured TTFB ~9.5 ms, total ~9.8 ms, no tail;
+`index.html` is ~1 KB.
+
+## Fix in place (option A — suppress the advert)
+
+Emit `Alt-Svc: clear`, which actively drops any cached alternative (better than merely
+deleting the header, which leaves the sticky 30-day cache in place):
+
+- **Prod / repo**: `deploy/caddy/Caddyfile` — a site-level `header Alt-Svc clear` (this
+  caddy terminates TLS in prod).
+- **Test contour**: the host caddy terminates TLS, so the fix lives there (homelab
+  config, outside this repo): `header Alt-Svc clear` on the `scrabble.*` site. The
+  in-compose caddy serves plain `:80` in test and never advertises h3, so the repo
+  directive is a harmless no-op there (the host caddy re-stamps the header).
+
+`header Alt-Svc clear` overrides Caddy's auto-advert (verified) and is site-scoped.
+
+### Verify
+
+The runner/prod host shell cannot reach the Docker bridge IPs directly, so probe from a
+container on the relevant network, using `--resolve` to hit the TLS-terminating caddy by
+its bridge IP (this also bypasses the public-IP NAT hairpin):
+
+```sh
+# <edge-ip> = the TLS-terminating caddy's IP on its network (docker inspect ... )
+docker run --rm --network edge curlimages/curl:latest -sS -D - -o /dev/null \
+  --resolve <host>:443:<edge-ip> https://<host>/telegram/ | grep -iE '^HTTP|^alt-svc'
+# expect: HTTP/2 200, and NO `alt-svc: h3=...` (the header is absent or `alt-svc: clear`)
+```
+
+## If it recurs — alternatives to try
+
+So we do not re-derive the diagnosis from scratch:
+
+1. **Re-confirm the advert is actually suppressed** with the verify command above. A
+   redeploy or a Caddy upgrade could regress it, or a client may still hold a cached
+   `h3` entry that has not yet been replaced by a `clear` (it needs one successful h2
+   response to receive the `clear`).
+2. **Option B — serve HTTP/3 for real** instead of suppressing it. Worth it only if we
+   actually want QUIC (the benefit is marginal for a ~1 KB shell plus hash-immutable
+   cached assets, and it adds UDP/QUIC attack surface):
+   - Publish UDP: add `"443:443/udp"` next to the TCP map in
+     `deploy/docker-compose.prod.yml` (and publish udp/443 on the test host caddy too).
+   - Open the firewall: add a `443 proto: udp` rule in
+     `deploy/ansible/roles/main/tasks/main.yml`.
+   - Drop the `header Alt-Svc clear` so Caddy advertises h3 again.
+   - Verify with an h3 client from inside the network:
+     `docker run --rm --network edge ymuski/curl-http3 curl --http3-only ...` should
+     return `http=3 code=200`.
+3. **Look past the edge** if the advert is suppressed and stalls persist. The h2 path is
+   fast server-side, so a remaining stall is most likely the client network / RTT / the
+   provider, not our stack. Re-run the timing loop (below) to confirm the server is
+   still <~10 ms TTFB before chasing the client side.
+
+## How this was diagnosed (method, to repeat)
+
+- The runner/prod host shell cannot reach the Docker bridge subnets, so all probing runs
+  from a throwaway container on the target network (`docker run --network <net>
+  curlimages/curl`), using `--resolve <host>:443:<edge-ip>` to bypass the public-IP NAT
+  hairpin and exercise the real TLS path.
+- Compare a fresh-connection timing loop (worst case, full TLS each time) against a
+  keepalive batch to separate handshake cost from serving cost:
+
+  ```sh
+  docker run --rm --network edge curlimages/curl:latest sh -c '
+    for i in $(seq 1 30); do
+      curl -sS -o /dev/null --resolve <host>:443:<edge-ip> \
+        -w "http=%{http_version} code=%{http_code} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
+        https://<host>/telegram/
+    done'
+  ```
+
+- `docker port <caddy>` shows whether `udp/443` is actually published; the response
+  `Alt-Svc` header shows what the edge advertises. The two disagreeing is the bug.