Posted Jul 2, 2026 at 4:29 PM UTCEExternal LinkCloudflare is cracking down on “multi-purpose” crawlers.Starting September 15th, Cloudflare will block bots that scrape ad-supported websites for search indexing and AI training at the same time. The goal is to encourage AI companies to separate their crawlers for different purposes, giving publishers more control over which ones to allow into their websites.Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.Emma Roth
Cloudflare is cracking down on “multi-purpose” crawlers.
Starting September 15th, Cloudflare will block bots that scrape ad-supported websites for search indexing and AI training at the same time. The goal is to encourage AI companies to separate their crawlers for different purposes, giving publishers more control over which ones to allow into their websites. [Link: Cloudflare's new policy pushes AI companies to pay for publishers' content | TechCrunch | https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/ | TechCrunch]
83 words~1 min read
