Cloudflare, which services about 20% of websites, announced Tuesday that it is now blocking web-scraping AI bots from accessing those sites by default.
Unless a website explicitly turns off the default, an AI crawler will need to obtain permission from the website to scrape its content. Website owners can choose whether they want AI crawlers to access their content, and decide how AI companies can use it, Cloudflare explained in a statement.
AI companies can now clearly state their purpose — whether their crawlers are used for training, inference, or search — to help website owners decide which crawlers to allow.
Cloudflare explained that for decades the internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators who produce high-quality content with financial compensation and a growing following, while helping users discover new and relevant information.
That model is now broken, it continued. AI crawlers collect content like text, articles, and images to generate answers, without directing visitors to the original source, thereby depriving content creators of revenue and the satisfaction of knowing someone is viewing their content. If the incentive to create original, high-quality content disappears, society loses, and the future of the internet is at risk.
