Benchmarks & methodology

Last measured 2026-09-06 (exact retrieval) · Full raw write-up ships in the package as BENCHMARKS.md

1. Search quality — 96/100 (human-scored relevance)

Methodology: 20 canonical queries covering the search patterns agents actually issue — exact function lookup, conceptual questions ("how does authentication work"), cross-file tracing, configuration discovery, error handling, template search, API endpoints, security mechanisms, infrastructure, legacy code. Each query's top-5 results were scored 0–5 for relevance by a human evaluator. 96/100 = an average of 4.8/5 across the 20 queries.

Corpus: a real production marketplace codebase — ~3,500 PHP/JS/Smarty files, ~11,400 indexed chunks, 20 years of accretion. Not a curated demo repo.

Cross-codebase validation: the same human-scored protocol on the public Laravel framework (36K chunks) scored 83/100 across 30 queries spanning middleware, Eloquent, queues, auth, caching, validation, mail, sessions, routing, and more.

Garbage-input robustness: 10 nonsense/off-topic queries (pizza recipes, random unicode, SQL injection strings). Three correctly returned zero results; the rest returned only low-confidence results below the actionable threshold. No false confidence.

2. Token efficiency — ~80% savings

Methodology: 12 real development tasks performed twice with Claude Code — once with plain grep/file-reading, once with CodeNavi as the retrieval layer — measuring context-window tokens consumed before reaching the correct solution.

The two no-difference tasks were simple, well-named function lookups where grep was equally effective.

3. What we fail — adversarial query shapes

A second, deliberately hostile 10-query set targets shapes the canonical 20 don't cover: caller-tracing, polysemous verbs, intent with zero vocabulary overlap, absence queries. Pure semantic search scored 19/50 (38%) on it.

This is why CodeNavi ships structural tools alongside search: find_callers, impact_analysis, and grep_search run against a call-graph sidecar built at index time. Re-running the four structural/flow queries with the real agent flow — search to find the entity, then a structural tool — moved that subset from 10/20 to 17/20. One cluster stays out of scope by design: absence queries ("where is X missing") are not a retrieval problem — use static analysis or linting for that shape. We won't promise it.

4. Automated regression scores (stricter, lower — published anyway)

Every release also runs an automated harness (tools/benchmark.py, ships in the package): 20 queries per codebase with an expected-substring answer key, scored by counting how many expected files/symbols land in the top-5. This measures coverage of a fixed answer set, not relevance — a top-5 full of correct, useful results still scores low if it surfaces different (equally valid) files than the key expects. It exists to catch regressions between releases, not to describe experienced quality; it is deliberately hard to please. Scores from the 2026-09-06 run on the live dev index, read with exact retrieval — the approximate vector index adds up to ±7 points of build-to-build noise, so the published numbers bypass it:

The MedusaJS corpus is scoped to packages/ since 2026-09-06: the repository's docs site (www/) had made 58 % of the chunks, and generated API docs were competing with product code for the same top-5 — the same dilution a monorepo with a docs subtree sees until that subtree is excluded at index time (EMBED_SKIP_DIRS).

Since 2026-09-05 every set also carries four identifier-bearing queries — a bare symbol such as Str::slug or bz_locations, or prose plus an identifier. All 20 hit the defining file; they are scored separately so the table above stays comparable with earlier runs. The same run is the committed baseline a release cut is gated against.

The human-scored 96/100 and the automated 68/100 on the same codebase are different instruments measuring different things — the gap between them is the difference between "were the top-5 results relevant?" (what a user experiences) and "did the top-5 contain these exact pre-listed substrings?" (what a regression gate needs).

5. Limitations — read these

• Human-scored numbers come from a single evaluator (the author). No inter-rater reliability.
• No head-to-head against competing code-search products on identical query sets yet.
• The adversarial set is one-codebase-specific; structural-query scores vary with how concentrated a codebase's handlers are.
• Retrieval quality depends on the embedder seeing your language mix — see the supported-language matrix before assuming parity with these corpora.

Want to check the numbers on your code instead of ours? Request a 30-day trial — the package includes the benchmark harness, so you can build your own query set and hold us to it.