Cloudflare gives all customers new tools to control how AI bots access their websites.
Since July 2024, Cloudflare customers have been able to block all AI crawlers with a single click. Now, instead of a blanket block, site owners can choose between three categories. Search covers search engine indexing. Training covers data collection for AI model training. Agent covers bots acting on behalf of users, like ChatGPT.
Cloudflare's new AI bot controls in the dashboard let site owners manage access for Search, Training, and Agent bots separately. | Image: Cloudflare
Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that carry ads, since advertising signals that a site wants human visitors. Search crawlers will still be allowed. Multi-purpose crawlers like Googlebot that both search and train will be treated according to whichever rule is strictest. The new options are available to users on the free plan as well.
Cloudflare is pushing bot operators to split their crawlers by purpose. Cloudflare CEO Matthew Prince has previously called out Google for bundling its search crawler with its AI crawler. In June 2026, Prince noted that bot traffic had surpassed human traffic on the internet for the first time, a milestone he originally didn't expect until late 2027.
