Observability Quick Start — dashboards in 2 minutes

A turnkey Prometheus + Grafana stack that scrapes a running CodeNavi server and auto-loads the dashboard. For the full reference (metric list, two scrape endpoints, panel descriptions) see observability.md.

Everything stays on-prem: the metrics are counts / histograms / gauges only — no query text, no code. Don't ship them to a third-party SaaS.

Prerequisites

• A running CodeNavi server exposing metrics on 127.0.0.1:9091 (the default CODENAVI_METRICS_PORT; confirm with curl -s http://127.0.0.1:9091/metrics | head).
• Docker + Docker Compose on the same host.

1. Bring up the stack

cd docs/observability/stack
GRAFANA_ADMIN_PASSWORD='<choose-a-real-password>' docker compose up -d

That starts: - Prometheus on http://localhost:9090 — scrapes 127.0.0.1:9091 (config: prometheus.yml), with the alert rules from alerts.yml loaded. - Alertmanager on 127.0.0.1:9093 — routes firing alerts (config: alertmanager.yml). - Grafana on http://localhost:3000 — with the Prometheus datasource and the CodeNavi — search & index observability dashboard already provisioned.

2. Open it

Browse to http://localhost:3000, log in as admin / the password you set, and open the dashboard (it's in the dashboard list, or search "CodeNavi"). Pick a codebase / tool from the $codebase and $tool dropdowns at the top.

3. Verify the scrape is healthy

Prometheus → Status → Targets (http://localhost:9090/targets): the codenavi target should be UP. If codenavi_index_chunks is queryable in the Prometheus expression browser, you're done.

Query-rate panels look flat? That's expected with no traffic — rate() only registers when the counters keep climbing. Run a few searches and the rate/latency panels fill in. The index, feedback, and server-health gauges show immediately.

Checking VRAM? Use the VRAM held by CodeNavi stat in Index & server health — that's the server process's own usage. The device used series next to it is the whole card including every other GPU client (on WSL2, that includes Windows), so it reads higher and moves on its own. All four read 0 on a CPU-only deploy.

Alerts

Four rules ship enabled in stack/alerts.yml, all built on metrics the server already exports:

Alert	Fires when	Severity
CodeNaviTargetDown	the metrics endpoint hasn't been scrapeable for 5 min	critical
CodeNaviReindexStale	no completed reindex for 26 h (catches a failing or silently-dead reindex trigger, e.g. a forge webhook)	warning
CodeNaviQueryErrorRate	>5% of queries error over 10 min	warning
CodeNaviSlowQueries	p95 total query latency >5 s over 10 min	warning

Firing alerts are visible at http://localhost:9090/alerts. Notifications need one edit: the shipped stack/alertmanager.yml has a placeholder receiver — replace it with your Slack/email/PagerDuty channel and restart the stack. Thresholds live in alerts.yml; adjust them to your reindex cadence and latency budget.

Prefer webhooks over Prometheus? A failed background reindex also emits a reindex.completed event with status:"failed" to HOOK_URL — the alert rules and the hook are independent paths to the same signal.

Docker Desktop (Mac / Windows)

Host networking isn't available, so:

1. On the CodeNavi server set CODENAVI_METRICS_BIND=0.0.0.0 (so the metrics port is reachable from a container) and restart it.
2. In docker-compose.yml, drop network_mode: host from both services, add ports: (9090:9090, 3000:3000), and add to each:

   extra_hosts:
     - "host.docker.internal:host-gateway"
   

3. In prometheus.yml, change the target to ["host.docker.internal:9091"].
4. In grafana/provisioning/datasources/codenavi.yml, set the url to http://host.docker.internal:9090.

Remote / authenticated scrape

To scrape over the API port with a bearer token instead of the co-located :9091, point the job at :8765 with metrics_path: /metrics and an authorization.credentials of your CODENAVI_AUTH_TOKEN — see the commented block in prometheus.yml and observability.md § Two ways to scrape.

Manual import (no stack)

Already have Prometheus + Grafana? Add the scrape job from prometheus.yml, then in Grafana go Dashboards → New → Import, upload grafana-dashboard.json, and pick your Prometheus datasource. The dashboard has a pinned uid (codenavi-observability), so re-importing overwrites in place instead of creating duplicates.

Security

• Set a real GRAFANA_ADMIN_PASSWORD (the compose file refuses to start without one).
• Keep Grafana/Prometheus bound to the host or your private network — the dashboard data is for your eyes, and the metrics endpoint itself is unauthenticated on :9091 (bind it to loopback, the default).