CodeNavi on rented GPUs (vast.ai, RunPod, Hyperstack)¶

Self-hosted CodeNavi assumes the customer controls the host. Renting GPU time from a marketplace like vast.ai breaks that assumption: the host operator has root on the box your model runs on. This page documents the validated recipe for shipping CodeNavi onto a rented GPU when you accept that trade-off, and the hardware floor below which indexing won't work.

Read the EULA Addendum first. Rented GPU deployments are governed by Addendum A — Rented GPU Deployments of the CodeNavi Commercial License Agreement. Verified-datacenter offers only; community/spot offers are out of scope. If you haven't signed the addendum, stop here.

When this applies¶

Use this guide if:

You can't or won't host CodeNavi on hardware you operate yourself.
You've signed Addendum A and chosen a verified-datacenter offer on a marketplace.
You've understood that the host operator can, in principle, copy the running model weights and the indexed code. The per-customer weight fingerprint² is a forensic mark, not a prevention mechanism.

If you can run on-prem, do that instead — none of this applies. A GPU VM in your own AWS or Google Cloud account is a different case with its own verified recipe: Cloud GPU VM.

Hardware floor¶

The Buric-E4B embedder (4 B parameters) is ~7.5 GiB resident in fp16 before a single chunk is encoded. Each encode batch's activations must fit in whatever VRAM remains. Because every chunk is capped at 1024 tokens (by the model's max_seq_length and by chunk-splitting⁵), a batch's activation is bounded by batch size, not file size. Runtime autotune⁵ picks the batch by total VRAM and loads the reranker on CPU on small cards — so the container self-tunes inside vastai create (which bypasses setup.sh) with no operator override. It also bakes PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which reclaims the allocator fragmentation (~720 MiB) that caused the original small-VRAM OOM.

Below ~10.5 GiB that fp16 residency simply does not fit, so the embedder loads int8-resident¹¹ instead (~4.1 GiB; weights dequantized per forward, embeddings bit-identical) and 8–10 GiB cards become usable at ~23% less indexing throughput. Measured on a 10.58 GiB 2080 Ti over a dense corpus: 5.41 GiB reserved / 12.3 s at the autotuned batch 2, against 8.28 GiB / 10.1 s for the fp16 path at batch 1. It is an availability floor, not a rental strategy — renting below 11 GiB to save cents and costs hours.

Workload	Min VRAM	Examples	Autotuned batch	Indexing speed
Indexing + serving (Tier A — recommended)	24 GiB	RTX 3090, RTX 4090, A6000, L40S	16	full default
Indexing + serving (Tier A — 16 GiB)	16 GiB	RTX 4080 / 4080 Super, RTX A4000	4	~2-3× slower
Indexing + serving (Tier A — 11–12 GiB)	11 GiB	RTX 2080 Ti, RTX 3060, RTX 4070	1	~6-8× slower
Indexing + serving (Tier A — int8-resident¹¹)	8 GiB	RTX 3070/4060, RTX 3080 10 GiB	2	~8-10× slower
Query-only (index pre-built elsewhere, Tier B)	8 GiB	any ≥8 GiB CUDA card	n/a	n/a

Architecture matters as much as VRAM — filter out pre-Turing. The GA image now ships the universal cu128 / torch 2.7.1 wheel¹², whose kernels cover sm_75 – sm_120: Turing through Blackwell — RTX 20/30/40/50-series, T4, A10/A10G, A100, L4, L40S, A4000/A6000, and the RTX PRO Blackwell cards are all supported. Pre-Turing cards — Volta (sm_70; V100, Titan V), Pascal (sm_60/61; P100, GTX 10-series), and Maxwell — are not supported (CN-568: NVIDIA's R580 driver branch was the last to carry them); the embedder fails loud at model load¹², naming the GPU's sm_XY, instead of silently 0-resulting. The cu128 wheels also require an NVIDIA driver ≥ 570.65 on the host. So when searching offers, filter out Volta/Pascal/Maxwell entirely; anything Turing-or-newer (A100, L40S, A6000, RTX 4090, RTX PRO Blackwell) is known-good. A per-arch cu124 build for a Turing/Ampere/Ada-only fleet is available on request — see gpu-matrix.md.

11–12 GiB is a supported indexing tier — there is no codebase-size limit. After the ~7.5 GiB model load these cards have ~3 GiB free; re-measured sustained encodes of a dense codebase on a 10.58 GiB RTX 2080 Ti (with expandable_segments) peak at ~10.5 GiB at batch 2 — OOMing an 11 GiB card on the first batch — so the tier uses batch 1 (~0.5 GiB margin¹⁰). Chunk-splitting keeps every chunk ≤1024 tokens, so even huge files index fine — the only cost at 11–12 GiB is throughput (~6–8× slower than a 24 GiB card), never capacity. The original 2026-06-17 RTX 4080 Super (16 GiB) OOM was the old default (batch 16 + reranker on GPU + allocator fragmentation); the batch-4 + reranker-CPU + expandable_segments tier fixes it.

The reranker (search-side) is comfortable everywhere ≥8 GiB — only the indexing encode stresses VRAM.

Sharing the GPU with other workloads (own-hardware mode). This page is mostly about rented GPUs, but the same trade-off applies on your own card if you also run Ollama, a generative model, or training jobs on it. Set CODENAVI_VRAM_BUDGET_GIB=<N>⁶ to make CodeNavi autotune as if the card only had N GiB — e.g. CODENAVI_VRAM_BUDGET_GIB=11 on a 32 GiB card forces the 11–12 GiB tier (batch 1 + reranker on CPU) and leaves the rest free. Soft cap only — it changes autotune decisions, not actual PyTorch allocation; a concurrent VRAM spike from the other workload can still cause OOM. Details in setup.md § GPU autotune overrides.

Disk: budget 35 GiB. The image is 14.4 GiB uncompressed; an indexed mid-sized codebase plus logs land around 18-22 GiB total.

Pull throughput on a verified vast.ai datacenter (Germany → Cloudflare R2) was ~100 MB/s on our test, so a cold pull of the full image takes 3-5 min. Subsequent instance creations on the same machine reuse the layer cache and come up in ≈20 s.

Recipe — vast.ai¶

0. Prerequisites¶

An offer ID for a verified datacenter machine with ≥11 GiB VRAM (≥24 GiB recommended for full throughput; 16 GiB is the comfortable mid-tier; 11–12 GiB works for any codebase but ~6–8× slower — see Hardware floor) and direct_port_count ≥ 4 (you'll publish 8765 + 9091, plus headroom).
Your CodeNavi image hosted on a private OCI registry under a domain you control, with htpasswd credentials.
Your signed license key.
vastai CLI authenticated against your account.

1. Find an offer¶

vastai search offers 'gpu_ram>=24000 verified=true rentable=true direct_port_count>=4' \
  --order 'dph_total'

Pin to a specific machine you've vetted with machine_id=<N> and host_id=<M>.

2. Critical: do not pass `--ssh`¶

The --ssh flag tells vast.ai to inject a /.launch wrapper as the container's ENTRYPOINT. That wrapper calls the ssh binary to establish a reverse tunnel to vast's relay. CodeNavi's image (Debian slim base, no openssh) crash-loops with /.launch: line 48: ssh: command not found and your real entrypoint never runs.

Use args launch mode instead — pass --entrypoint and --args. Per vast.ai's own docs: "If you use args/entrypoint launch mode, we create a container from your image as is, without attempting to inject ssh and or jupyter."

3. Bootstrap entrypoint with inline license¶

There's no usable vastai execute channel without ssh in your image. The workable pattern is a /bin/bash entrypoint that writes the license file then execs CodeNavi as PID 1 — and runs cleanly across any restart (the opt-in CODENAVI_REINDEX_RESTART=1 post-reindex restart, or a watchdog fallback) because it's the container ENTRYPOINT and re-runs on every container start.

B64=$(base64 -w0 ./license.key)               # your signed license key
TOKEN=$(openssl rand -hex 16)                  # ephemeral CODENAVI_AUTH_TOKEN

vastai create instance "$OFFER_ID" \
  --image  <your-registry>:5443/<repo>:<tag> \
  --login  "-u <customer-id> -p <reg-password> https://<your-registry>:5443" \
  --disk   35 \
  --direct \
  --env    "-p 8765:8765 -p 9091:9091 -e CODENAVI_AUTH_TOKEN=$TOKEN" \
  --entrypoint /bin/bash \
  --args -c "set -e; mkdir -p /app/data; printf %s '$B64' | base64 -d > /app/data/license.key; chmod 600 /app/data/license.key; exec python server.py --http"

Notes:

--direct is what publishes the container ports to the offer's WAN IP. Without it, the docker port mapping is internal-only and outside traffic gets connection refused.
The bootstrap inlines the license as base64 (a Pro license is ~500 bytes, comfortably inside --args).
The exec in the shell command means PID 1 becomes python server.py --http, not bash — so SIGTERM from vast.ai cleanly stops CodeNavi.
The --args flag must be the last argument; vast.ai parses everything after it as the args string.

4. Verify the deployment¶

After vastai show instance <id> reports actual_status=running:

# Read host:port from `vastai show instance` — the value under ports[8765/tcp].HostPort
HOST=$(...); PORT=$(...)

# Health (unauth) — should be 200 with version + license tier
curl -sS http://$HOST:$PORT/health

# UI (unauth) — should be 200 with the SPA HTML (~20 KB)
curl -sSI http://$HOST:$PORT/ui

# MCP without auth — should be 401
curl -sS -o /dev/null -w '%{http_code}\n' http://$HOST:$PORT/mcp

# MCP with auth + index_stats — if you placed the license correctly, you'll see
# "No index exists. Call reindex first." (license unlocked, no codebase yet).
# If you see "CodeNavi requires a valid signed license …", the bootstrap didn't
# write the license to /app/data/license.key — fix and recreate.
curl -sS -H "Authorization: Bearer $TOKEN" \
         -H 'Content-Type: application/json' \
         -H 'Accept: application/json, text/event-stream' \
         -X POST http://$HOST:$PORT/mcp \
         -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"index_stats","arguments":{}}}'

Container logs are available even without ssh:

vastai logs <instance-id>

Look for: Code embedding model warmed up (local), Reranker warmed up (ms-marco), and the bootstrap ls -l /app/data/license.key line confirming the key landed at mode 600.

5. Get your codebase onto the box¶

Two viable patterns; pick by codebase sensitivity:

Customer-controlled fetch: pull the codebase in the bootstrap script. The GA image ships no git/curl/wget (only python + tar), and the container runs as uid 999 which cannot mkdir at / — so fetch with Python stdlib and extract to an appuser-writable path (/app/codebase, not /codebase):
```
python -c "import urllib.request; urllib.request.urlretrieve('<tarball-url>','/tmp/c.tgz')"
mkdir -p /app/codebase && tar xzf /tmp/c.tgz -C /app/codebase --strip-components=1
export CODEBASE_CONFIG='{"<name>":"/app/codebase"}'
```
Treat any embedded deploy key / token as exposed — see Trust model below.
Pre-built index, no source: index the codebase on hardware you control, then ship only the LanceDB tables onto the rented box. Search works, reindex does not. This is the recommended pattern for sensitive code — see cloud-reindex.md for the broader same-shape recipe.

Trust model¶

Anything that lands inside the rented container is in principle readable by the host operator with root access:

The Buric-E4B weights (your per-customer fingerprint identifies leaks but does not prevent them²³).
The license key (re-issuable per Addendum A § 4 if compromised). Note that the bootstrap-entrypoint pattern in §3 above inlines the base64-encoded license into the container's Cmd — the host operator can recover the license via docker inspect even before it ever lands in /app/data/. Treat each rented engagement's license as disclosed-by-deployment and plan for rotation on engagement end.
The CodeNavi source code (compiled to .so for IP-bearing modules¹⁴ — but not weight-protection).
Your indexed source code, if you ship it to the box.
The secret scanner ships DISABLED on a bare image. With no data bind-mount (the args-launch recipe above), the image bakes no secret_rules.json[.example], so the scanner loads ZERO rules — secret redaction and injection detection are off, and indexed code (including secrets) is embedded unredacted. On a build that doesn't bake the rules into the image⁷, seed /app/data/secret_rules.json in the bootstrap before starting the server. (Compose/trial deployments are unaffected — they bind-mount data/.)

Treat every rented instance as compromised by default. Use a per-engagement registry password and rotate it after each engagement. Use a fresh CODENAVI_AUTH_TOKEN per instance. Request a fresh license per engagement and treat the prior one as disclosed-by-deployment — there is no revocation mechanism yet⁸ (offline licenses stay valid until expiry), so keep expiries short.

Common mistakes¶

Symptom	Cause	Fix
`vastai logs` shows `/.launch: line 48: ssh: command not found` in a loop	You passed `--ssh` against an image without openssh	Recreate without `--ssh`; use `--entrypoint` + `--args`
All exposed ports refuse connections from the WAN IP	`--direct` not passed	Recreate with `--direct`
`/health` works, `index_stats` returns "CodeNavi requires a valid signed license…"	License didn't land at `/app/data/license.key`	Verify base64 encoding; check `vastai logs` for the bootstrap `ls -l` output
Reindex starts, then container restart + 0 chunks + lots of "errors"	CUDA OOM — VRAM too small (an image predating the VRAM autotune⁵ at default batch on <24 GiB)	Use a current image (autotune handles <17 GiB) or pick a ≥24 GiB offer
`vastai execute …` returns `400: Invalid command given`	Channel needs ssh in image	Use the bootstrap entrypoint pattern instead
`mkdir: cannot create directory '/codebase': Permission denied` crash-loop	Bootstrap writes the codebase to a path the appuser (uid 999) can't create	Use `/app/codebase` (appuser owns `/app`), not `/`
After `vastai update instance --args`, container logs `/bin/bash: c: No such file or directory`	`update --args` corrupts a complex `-c "…"` string (drops the leading `-`)	Don't use `update` for launch args — `destroy` + `create` (REMAINDER preserves them). To keep the cached image, create on the same `machine_id` (~30 s vs a cold pull).
Boot log: `secret scanner loaded ZERO rules … DISABLED`	Bare image, no `data` bind-mount, image bakes no `secret_rules.json`⁷	Seed `/app/data/secret_rules.json` in the bootstrap, or use a build that bakes the rules
Container restarts mid-reindex but recovers cleanly	Only with `CODENAVI_REINDEX_RESTART=1` (opt-in). The default post-reindex VRAM cleanup is an in-process reset⁹ that keeps the server up — no restart	Nothing to fix; if you set the restart fallback, the bootstrap re-applies the license on next start
Prometheus metrics unreachable from outside	Binds 127.0.0.1 by design	Run a scrape sidecar inside the container, or open a separate ticket