galaxy-game/tools/dev-deploy/monitoring

tools/dev-deploy/monitoring/ — observability stack

The long-lived dev environment runs a full metrics + logs + traces stack alongside the application as a production mirror: the same compose fragment and collector configs are meant to back production later. Every collector lives on the internal galaxy-dev-internal network and publishes no host port. The browser-reachable pieces (Grafana and the Mailpit UI) sit behind the operator console's single /_gm Basic Auth gate — see ../README.md and ARCHITECTURE.md §14.

Services

Service	Image	Role	Reachable
galaxy-prometheus	prom/prometheus	Scrape + store metrics (15d)	internal :9090
galaxy-loki	grafana/loki	Log store (7d)	internal :3100
galaxy-promtail	grafana/promtail	Ship container logs to Loki	—
galaxy-tempo	grafana/tempo	Trace store (3d), OTLP receiver	internal :3200, OTLP :4317/:4318
galaxy-node-exporter	prom/node-exporter	Host metrics	internal :9100
galaxy-cadvisor	cadvisor	Per-container CPU/memory/IO	internal :8080
galaxy-grafana	grafana/grafana	Dashboards + Explore	Caddy /_gm/grafana/

What is collected

• Metrics. Prometheus (30s interval) scrapes the backend Prometheus endpoint (galaxy-backend:9100), the gateway admin endpoint (galaxy-api:9191), node-exporter (host) and cAdvisor (per container). Engine containers expose no /metrics; cAdvisor covers their resource use.
• Logs. promtail discovers containers through the Docker API, filtered to the galaxy.stack=dev-deploy label, and ships their stdout/stderr to Loki labelled by container.
• Traces. backend and gateway export OTLP traces over gRPC to Tempo (galaxy-tempo:4317), plaintext on the internal network (OTEL_EXPORTER_OTLP_INSECURE=true, since Tempo's receiver is not TLS-wrapped inside the contour).

Grafana access (behind the /_gm gate)

Grafana is served under /_gm/grafana/ (GF_SERVER_ROOT_URL + GF_SERVER_SERVE_FROM_SUB_PATH=true) behind the shared operator gate: the Caddy /_gm/* Basic Auth (the admin-console account) is the only barrier. Grafana itself runs as anonymous Admin with its login form and basic auth disabled (GF_AUTH_ANONYMOUS_ENABLED=true, GF_AUTH_ANONYMOUS_ORG_ROLE=Admin, GF_AUTH_DISABLE_LOGIN_FORM=true, GF_AUTH_BASIC_ENABLED=false), so it ignores the forwarded credentials and asks for no second password. GALAXY_DEV_GRAFANA_ADMIN_PASSWORD still seeds the admin user for provisioning/API use.

Datasources (Prometheus, Loki, Tempo) and a starter dashboard (grafana/dashboards/galaxy-overview.json) are provisioned as code under grafana/provisioning/.

Config delivery

dev-deploy.yaml copies this directory to a stable host path ($HOME/.galaxy-dev/monitoring, exported as GALAXY_DEV_MONITORING_DIR) before compose up, and the compose binds it read-only into the collectors. A stable path — not the ephemeral CI workspace — keeps the mounts valid across container restarts and host reboots (the same lesson as the geoip volume; see ../KNOWN-ISSUES.md).

Tuning (cost knobs)

Defaults favour the smallest workable footprint; all are config/compose values:

• Prometheus scrape_interval=30s, --storage.tsdb.retention.time=15d.
• Loki retention_period=168h (7d); Tempo block_retention=72h (3d).
• cAdvisor --housekeeping_interval=30s.
• Per-service deploy.resources.limits.memory caps (~1.5 GB total cap; steady-state well under that).

Seven always-on containers cost roughly ~1.1 GB steady RAM and ~1.5–2.5 GB disk at these retention windows. cAdvisor is the main CPU cost; on a constrained host it can be dropped (host + app metrics still cover most needs).