Microsoft has one big piece of advice for publishers and retailers that are cautiously navigating the AI era: Let the bots scrape your sites.
That’s according to Nikhil Kolar, VP of publisher product at Microsoft AI, who spoke at AdExchanger’s Programmatic AI event in Las Vegas last week.
Rather than fighting against the rising tide of AI-based search engines and agentic tools, Kolar said publishers and retailers should actually be creating content that can speak directly to these bots (and thus the LLMs generating new forms of traffic). And site owners should also update their robots.txt files to avoid restrictions on crawlers that will diminish the site’s visibility to AI engines.
Currently, though, four out of five websites block traffic from AI bots and crawlers, according to Kolar. As a result, “the content or products that you have are not legible to agents.” Which means, he added, that “your business is closed” and “you get no discovery, no recommendations that you’re a part of and no demand.”
Leaning on licensing
