Your fetch agent knows two endings to a request. 200: parse it. 403: back off, rotate, or skip. That branch has been the whole game for years.
There's a third ending now, and it's the one your code falls through. 402 Payment Required, with a dollar amount in the header. Cloudflare turned it on for Pay-Per-Crawl in July 2025. 403 punished you with retries — wasted time, nothing you couldn't see. A bare 402 isn't a charge by itself; it's a quote. But the moment your agent does the obvious thing — re-request and agree to the price — it's an invoice. And here's the part that bites: by default, your HTTP client has no brake for it.
TL;DR
Cloudflare Pay-Per-Crawl answers crawlers with HTTP 402 Payment Required plus a crawler-price header. To pay, the crawler re-requests with crawler-exact-price and expects 200. (Cloudflare, 2025-07-01)
On a synthetic 10-host fixture (prices I made up to exercise every branch), a naive agent that treats 402 like "just pay and move on" spent $0.9658 against a $0.10 budget — roughly 10x over budget, or 16x what the budgeted agent spent ($0.0600). Same numbers, two different baselines; I keep them straight below. It fetched 9 of 10 pages.
