Deploy crawler¶

The crawler walks a running site — staging or production — and stores every page, redirect and error it meets in a local run history. Run it after each release and diff the run against the last clean one: new errors surface as a list, before a customer or a partner reports them. All history stays on the host that runs CodeNavi.

It is off by default and adds nine tools to the agent surface when on, so turn it on only on an install that watches a site.¹

Enable it¶

In .env, next to docker-compose.yml:
```
CRAWLER_ENABLED=true
CRAWL_PROD_BASE_URL=https://www.example.com
CRAWL_DEV_BASE_URL=https://staging.example.com
```
CRAWL_LOCAL_BASE_URL covers an env=local target; note it resolves inside the container, so a site on the host machine is http://host.docker.internal:<port>.
Recreate the container so the environment change reaches it:
```
docker compose up -d --force-recreate
```
A plain restart keeps the old environment.
The Crawler screen appears in the web UI, and the crawler tools appear in the agent's tool list — see Crawler tools.

Run a crawl¶

From the web UI: Crawler → Operate, pick the environment (local, dev, prod) and full or changed. From an agent: the crawl tool. From a shell:

./codenavi crawl run prod
./codenavi crawl status

A run records every URL fetched, its status, redirects and content type, plus errors grouped by type (HTTP 5xx, timeouts, connection errors, session expiry).

Compare two runs after a deploy¶

compare_runs takes two run ids and reports what changed: errors that are new since the earlier run, and errors that disappeared. That diff is the deploy report. The Crawler screen has the same comparison under Forensics, and search_errors, search_urls and get_referrers search the history in plain language ("checkout 500s this week") or by link relationship (every page that links to a URL you are about to retire).

Be gentle with the target¶

Defaults are four parallel requests with no delay. Most application servers behind a small worker pool answer that with 503s. Set per-environment limits in data/settings.json:

{
  "CONCURRENCY": { "prod": 2, "dev": 2 },
  "REQUEST_DELAY_MS": { "prod": 300, "dev": 300 },
  "CRAWL_MAX_RPS": { "prod": 4 }
}

Start at one or two parallel requests on a PHP-FPM or similar pool and raise it once a run finishes without 503s. The crawler honours 429 / Retry-After on its own. CRAWLER_EXCLUDE_URL_PATTERNS prunes faceted navigation and pagination from the walk — see Configuration.

The crawler does not log in. It sends a session cookie you give it, so the session is one you created with a service account in a browser.

Copy data/auth.ini.example to data/auth.ini and set its mode to 600. It is never committed and never leaves the host.
One section per environment; one line per privilege level. The value is the cookie exactly as the browser sends it, name=value:
```
[auth_prod]
member_cookie = SESSIONID=3f9a…
admin_cookie  = SESSIONID=71c0…
```
A second line <level>_mfa = name=value carries a second cookie when the site sets one after multi-factor sign-in.
Tell the crawler what a login page looks like on your site, and where to probe for a live session, in data/settings.json:²
```
{
  "CRAWLER_LOGIN_URL_PATTERNS": ["/login", "?redirect=login"],
  "CRAWLER_AUTH_PROBE_PATHS": { "member": "/account", "admin": "/admin/dashboard" }
}
```
- CRAWLER_LOGIN_URL_PATTERNS — substrings. A fetch whose final URL contains one, when the original URL did not, is recorded as a session-expiry redirect (auth_issue) instead of a normal page. Without this list, an expired session is crawled as a login page and looks like a healthy site.
- CRAWLER_AUTH_PROBE_PATHS — a path per level, appended to the environment's base URL. Before a crawl the crawler fetches it with the level's cookie and stops with a clear error if it lands on a login page.

Sessions expire. When a run reports auth_issue errors across the board, refresh the cookie in auth.ini and run again.

Safety¶

Crawling a logged-in session can reach links that change state. A compiled block list refuses destructive actions and cannot be switched off; an optional data/crawl_safety.json extends or relaxes it — see Crawler safety config. The crawler also refuses private and cloud-metadata addresses unless the environment is local (CODENAVI_CRAWLER_SSRF_GUARD).

Vendor change reference CN-174. ↩
Vendor change reference CN-690. ↩