This week’s Media Briefing looks at how AI crawlers now reach publisher content, one year after Cloudflare’s default bot changes.
The AI crawling environment for publishers is fragmenting. On one side is a visible layer of declared “good bots” that go through infrastructure gatekeepers like Cloudflare, Fastly and Akamai. On the other is a much larger gray crawling economy that still sits largely outside meaningful oversight.
Last week, Cloudflare rolled out new rules for “mixed crawlers” — bots that can be used both for basic search and for AI training or agentic tasks — declaring a “deadline to end the free pass” for these kinds of bots, which it says disadvantages site owners and more transparent AI companies. On Sept. 15, the defaults will be set to allow for search but block training and agent use for pages with ads.
That update lands a year after Cloudflare switched its default settings to make it easier for publishers to block AI crawlers, and underlines how quickly the crawling landscape has shifted from a blunt allow/deny decision to a layered mix of declared, mixed-use and gray bots. In practice, that now includes tackling the mixed-use giants whose crawlers have historically been treated as pure search, including Google.
