As AI bots flood their sites, publishers are no longer just trying to keep them out – they’re starting to re-engineer content for them. In doing so, publishers are preparing for an agentic web – a future in which AI agents make decisions and perform tasks on behalf of users. To stay visible in AI search, Time, The Economist and another major news publishers are already experimenting with parallel, agent-readable versions of their sites, stricter controls on which bots can crawl them and, in some cases, entirely new web standards.
Time, for example, is converting all its webpages from HTML into markdown versions, a simplified format that AI systems and agents can process more efficiently. Delivering content to AI agents in HTML is inefficient because it bundles layout, style and navigation information designed for humans and browsers. Markdown strips that away, giving agents the content and metadata, not the surrounding page design.
Last month, Time decided to block all AI bots by default, then created whitelists of approved bots that can access its content. Those approved bots are redirected to markdown versions of all of Time’s webpages, according to Mark Howard, chief operating officer at Time.
