Search quality¶
Air-gapped deployment is why CodeNavi exists; retrieval quality is why it's worth running. The quality is measured, published — misses included — and engineered, not incidental. Full methodology and query sets: codenavi.com/benchmarks.
The headline numbers¶
| Measure | Result | Protocol |
|---|---|---|
| Retrieval quality | 96/100 | 20 canonical queries on a production marketplace codebase (~3,500 PHP/JS/template files), top-5 results human-scored 0–5 for relevance |
| Cross-codebase validation | 83/100 | 30 queries on the public Laravel framework (36K chunks) — middleware, Eloquent, queues, auth, Blade, and more |
| Context-token savings | ~80% | 12 real development tasks run with and without CodeNavi, measuring agent context consumed before reaching the correct solution |
What that means for an agent: a question like "where do we handle token expiration?" comes back as the five chunks that matter, each with file:line — instead of the agent burning most of its context window reading files to find things.
No false confidence¶
Ten nonsense and off-topic queries (recipes, games, injection strings, random unicode) were run against the index: three correctly returned zero results, and the rest returned only low-confidence results below the actionable threshold. Scores are relative rankings, not calibrated confidence — the score guide makes low confidence visible to the agent rather than dressing it up.
Honest limits — and what covers them¶
We also publish the adversarial benchmark we don't ace: 10 queries picked for shapes plain vector retrieval is bad at — caller traces, polysemous verbs, intent with no vocabulary overlap, absence questions. semantic_search alone scores 38% on that set. Two things follow:
- Structural questions get structural tools. The real agent flow —
semantic_searchto find the entity, thenfind_callers/impact_analysis/grep_searchfor the structural part — lifted the structural/flow subset from 10/20 to 17/20 on re-measure. That's why the call-graph tools exist and why the bundled agent rules route those question shapes to them. - Absence questions are out of scope by design. "Which endpoints don't check auth?" is a static-analysis question; retrieval finds presence, not absence, and we don't promise otherwise.
Structural accuracy holds across languages¶
Retrieval scores move with the codebase — 83/100 on Laravel against 61/100 on Bugzilla, and we publish both rather than quoting the best one. Call-graph accuracy behaves differently: it stays put.
Measured 2026-09-10 on two public codebases, with ground truth taken from the
source before the tool was asked. Twelve symbols per codebase, each with a
single definition and two to eight call sites, sampled with a fixed seed from a
pool of several hundred; the test is whether find_callers returns the files
that genuinely contain the call sites.
| Codebase | Language | Caller relationships | Recall |
|---|---|---|---|
| Laravel | PHP | 39 | 100% |
| Medusa | TypeScript | 33 | 88% |
The same protocol run against a private legacy PHP application — procedural code named after the business rather than the mechanism — landed in the same band.
Why this matters when comparing tools: an AST call graph is easy to build badly and hard to build evenly. A tool can trace one language well and another barely at all, and a benchmark on somebody else's stack will not tell you which you are buying. If blast radius is why you are here, run the check above on your own repository — pick ten functions you already know the callers of, and count.
Where the quality comes from¶
- Buric-E4B — a code-specialized 4B-parameter embedder; wiring a generic embedder API into the same pipeline measured ~30 points worse.
- Cross-encoder reranking — every candidate is re-read against the query before results are returned.
- AST-aware chunking — functions and classes split on real boundaries across 15 languages, with oversized ones split into overlapping ~1,024-token windows so long functions aren't silently truncated at embedding time (a change that alone recovered 12 points on the Laravel suite).
- Vector-first by measurement — BM25 score fusion was benchmarked and removed; see Which RAG architecture is CodeNavi?
- A regression harness — an automated multi-codebase benchmark (stricter, substring-scored) gates changes, so quality claims are re-measured rather than remembered.
Judge it on your own code¶
The numbers above are our codebases. Yours is the benchmark that matters: index it, run your ten hardest "where is…" questions, and compare against grepping. The trial exists for exactly that.