Come 15 September, multipurpose crawlers used by the likes of Google, Microsoft and Apple will be blocked by default according to Cloudflare's new rules.

Cloudflare gave AI crawlers a September deadline to separate the bots that gather content for search from those that harvest it for training or be blocked on pages that carry ads.

Some crawlers gather data for both search and AI training, so when publishers block them to protect content they risk disappearning from search results ...