Skip to content

Observability — Prometheus + Grafana

CodeNavi ships prometheus_client instrumentation (metrics.py) covering query traffic, latency, retrieval quality, feedback, and index health. This guide wires it into a self-hosted Prometheus + Grafana so you can watch a CodeNavi server over time.

Just want it running? A turnkey Prometheus + Grafana stack (auto-provisioned datasource + dashboard) is in observability/stack/ — see observability/quickstart.md. This page is the full reference.

Everything here is on-prem: the metrics are counts / histograms / gauges only — no query text and no code ever leave the process — so they are safe to scrape into your own monitoring stack. Do not ship them to a third-party SaaS observability vendor.

Two ways to scrape

Both expose the same registry (same process), so pick one:

Endpoint Port Auth Use when
prometheus_client metrics server 9091 (CODENAVI_METRICS_PORT) none — reachable on the host's loopback only Prometheus is co-located with CodeNavi (same host / Docker network). The standard path.
GET /metrics 8765 (the API port) bearer token (CODENAVI_AUTH_TOKEN) Prometheus is remote and you want an authenticated scrape over the existing API port.

Because prometheus_client metrics live in the server process and the benchmark harness (tools/benchmark.py) runs in a separate process, these metrics reflect real server traffic only — benchmark runs never pollute them.

Prometheus scrape config

Co-located (port 9091, no auth):

scrape_configs:
  - job_name: codenavi
    static_configs:
      - targets: ["127.0.0.1:9091"]   # host.docker.internal:9091 if Prometheus is containerized

How :9091 stays private. CODENAVI_METRICS_BIND controls the bind inside the container, and the compose files set it per network mode — don't "tidy" it away:

  • Bridge mode (the default, incl. docker-compose.prod.yml): the container binds 0.0.0.0, and the ports: entry publishes it to the host's 127.0.0.1 only. The publish is what makes it private, not the container bind. Binding loopback inside the container would forward the published port to a listener that refuses the connection — which is exactly why /metrics was silently unreachable on every default install on older builds.1
  • GPU overlay (network_mode: host): the container's bind is the host's bind and ports: is a no-op, so it binds 127.0.0.1 — a 0.0.0.0 bind there would expose the unauthenticated metrics port on every host interface.

Scraping from another machine? Use the authenticated GET /metrics on 8765 below rather than opening 9091.

Remote / authenticated (port 8765, bearer token):

scrape_configs:
  - job_name: codenavi
    metrics_path: /metrics
    static_configs:
      - targets: ["codenavi.internal:8765"]
    authorization:
      credentials: "<CODENAVI_AUTH_TOKEN>"

Confirm the target is healthy at http://<prometheus>:9090/targets (state UP), then codenavi_queries_total should be queryable in the Prometheus expression browser.

Grafana dashboard

Import observability/grafana-dashboard.json:

  1. Grafana → Dashboards → New → Import.
  2. Upload the JSON (or paste it).
  3. Pick your Prometheus data source when prompted (DS_PROMETHEUS).

The dashboard is organized into five collapsible sections, most important first:

  • Index & server health — index chunks/files, time since last reindex (green <1h / yellow <24h / red ≥24h), avg reindex duration, index size by codebase, then two VRAM panels (a stat for what the server itself holds and a time series comparing that against the whole card — see below) and process health from the prometheus_client default collectors: resident memory (RSS), open file descriptors, Python GC collections/sec.
  • Traffic & quality — query rate by tool, error ratio, zero-result %, zero-result rate, safety-gate filtered chunks, alias expansions, and ranking feedback (cumulative submit_feedback counts: chunks promoted vs flagged as noise — agents flag several off-target chunks per query, so the second bar is normally the longer one; it is training signal, not a satisfaction score).
  • Latency & locks — total latency (p50/p95/p99), latency by phase (search/rerank/total), lock-wait p95.
  • Per-codebase — query rate by codebase, query rate by codebase × tool.
  • Compliance — compliance gap reports on disk, one row per pack × codebase.

Two template variables scope the panels (both single-select, All by default):

  • $codebase — sourced from codenavi_index_chunks so it lists every indexed codebase (not just ones with recent traffic).
  • $tool — the query tools (semantic/file/chat/symbol/refs/overview).

Reading the VRAM panels

codenavi_gpu_memory_bytes carries two different measurements, and the gap between them is the point:

  • torch_allocated / torch_reserved — this server process only. This is the answer to "how much VRAM is CodeNavi using". Nothing else can tell you: nvidia-smi --query-compute-apps returns [N/A] under WSL2, so per-process attribution is otherwise unavailable. reserved is what the caching allocator holds from the driver; allocated is what tensors actually occupy inside it. Reserved ≥ allocated always, and a large gap means cached-but-free blocks, not a leak.
  • used / total — the whole card, every CUDA client. On a shared GPU this includes everything else; on WSL2 the card is shared with the Windows host, so used can swing by gigabytes with zero CodeNavi activity. Don't read used as CodeNavi's footprint.

All four read 0 when there's no CUDA device, so total == 0 means "no GPU", not "a GPU with no memory". They're sampled at scrape time (set_function), so both scrape paths — the loopback :9091 exporter and the bearer-gated /metrics route — return live values.

Watch torch_allocated during a reindex to size the peak: a steady-state idle figure is much lower than the indexing peak, and the peak is what has to fit on the card.

They apply to the query/latency panels. The server-health, index, and feedback gauges are process-wide (no codebase/tool label), so they ignore the selectors. A Reindex annotation overlays vertical markers on the time-series panels at each completed reindex (from changes(codenavi_last_reindex_timestamp)), so latency/traffic shifts can be correlated with reindexes.

Metric reference

Metric Type Labels
codenavi_queries_total counter tool, status, codebase
codenavi_query_duration_seconds histogram tool, phase, codebase
codenavi_zero_results_total counter tool, codebase
codenavi_safety_gate_filtered_chunks_total counter —
codenavi_alias_expanded_total counter —
codenavi_feedback_votes gauge rating
codenavi_gpu_lock_wait_seconds, codenavi_search_lock_wait_seconds histogram —
codenavi_gpu_memory_bytes gauge kind (used/total = whole card, torch_allocated/torch_reserved = this process)
codenavi_index_chunks, codenavi_index_files gauge codebase
codenavi_last_reindex_timestamp gauge —
codenavi_index_integrity gauge codebase (1 ok, 0 mismatch, -1 unverified)
codenavi_reindex_duration_seconds histogram —

Both scrape endpoints serve this same set — see Two ways to scrape above. Descriptions and label meanings travel with the metrics themselves: curl either endpoint and read the # HELP lines.

Label values: - tool — every MCP tool is metered2, not just search. The six search tools keep their short, dashboard-stable labels: semantic, file, symbol, refs, overview, and chat (the LLM-grounded Q&A path; counted separately from direct search). Every other tool is metered by the central wrapper under its full name — grep_search, read_file, list_codebases, find_callers, impact_analysis, diff_impact, and the reindex/backup/crawler family — so a tool can no longer ship "dark" (a test enforces it). status is ok / error. - codebase — the queried codebase name (e.g. cellpex, laravel). On the query metrics, cross-codebase / unscoped queries are labelled all, and a codebase only appears once it has been queried since the last restart (counters are in-process). On the index gauges (codenavi_index_chunks / _files) every indexed codebase gets a series, set at startup and after each reindex — independent of query traffic. Low cardinality, so sum by (codebase) is cheap. The bundled dashboard's $codebase template variable is sourced from codenavi_index_chunks, so it lists every indexed codebase (not just ones with recent traffic), plus per-codebase query/latency/index-size panels.

Notes

  • Counters reset on restart — they live in process memory; Prometheus handles counter-reset semantics, so rate() / increase() stay correct across restarts.
  • Index gauges (codenavi_index_chunks / _files, _last_reindex_timestamp) are set at reindex time (and at startup).
  • The query log (deploy/logs/queries.log, behind the search_analytics tool) is a separate surface and is benchmark-dominated — it is not the basis for these dashboards; the search_analytics MCP tool documents its own parameters. Every tool also writes a JSON line here2: the search tools keep their rich per-tool records, and all other tools emit a {"message":"mcp_tool", tool, status, codebase, total_ms, client, ts_utc, ...} line. search_analytics reads only semantic_search / file_search records, so the mcp_tool lines don't skew query analytics.
  • Client attribution2 — best-effort identity of the calling MCP client, carried on every record as three fields. client is resolved in order: (1) the session's initialize clientInfo when stateful sessions are on (CODENAVI_STATELESS_HTTP=false); (2) the clientInfo recovered from the initialize handshake — under the default stateless transport the handshake identity isn't carried into a tools/call, so the scope middleware captures it from each initialize and caches it keyed by (ip, ua), letting a stateless tool call still be stamped with the real name (antigravity-client, Windsurf, the codex client) instead of a generic or absent User-Agent; (3) the raw User-Agent; (4) unknown. Recovery is best-effort: two clients that share an (ip, ua) collide, the later initialize winning. ua is the raw User-Agent and ip the source IP (first X-Forwarded-For hop when proxied, else the ASGI peer host) — both session-free, so they populate under stateless too; behind a reverse proxy / port-forward without X-Forwarded-For, ip is the proxy's address, not the client's. ts_utc is an explicit-UTC (Z) stamp, since asctime is the container's unlabeled local time — correlating it against a client log in another timezone otherwise silently mismatches.

  1. Vendor change reference CN-355. ↩

  2. Vendor change reference CN-349. ↩↩↩