Come 15 September, multipurpose crawlers used by the likes of Google, Microsoft and Apple will be blocked by default according to Cloudflare’s new rules.
IT and network services provider Cloudflare has announced new rules designed to give website owners more control over the types of web crawlers that will be allowed or blocked from their sites – along with plans to block multipurpose crawlers by default on ad-supported pages.
Traditionally, search engines and websites maintained a sort of “symbiotic relationship”, as Cloudflare puts it, whereby web owners allowed search engines to crawl their sites and in return, search engines sent users back to their pages.
The company explained that this crawl-to-referral process, when balanced, would help sites generate the pageviews needed to sustain advertising, affiliate revenue and subscriptions.
However, the rise of AI crawlers and agents changed things, where AI chatbots scrape sites to synthesise answers and bypass original sources – often leading to imbalanced crawl-to-referral ratios. Cloudflare’s own research from last year noted ratios of 118:1 up to nearly 50,000:1 – meaning an AI crawler could have scraped a site thousands of times and only sent back a single user.
