AI agents often begin with a deceptively simple web-access loop: take a URL, fetch it, extract text, and pass the result to a model. That loop works in a demo. In production, it can become a source of latency spikes, runaway costs, repeated requests, and inconsistent evidence.
A crawl budget is the control system that keeps this work predictable. It is more than a request limit. A useful budget decides which pages deserve attention, how much effort each page may consume, and when the agent should stop.
Start with the job, not the crawler
The correct budget depends on the agent's task. A monitoring agent may revisit a small set of pages on a schedule. A research agent may explore many domains once. A shopping agent may need current prices but can ignore most navigation pages.
Write the task as a small contract before choosing limits:
