'We don't serve their kind here' protects today's publishers, but won't help rebels destroy the monetization Death Stars built by Google and Meta
OPINION Cloudflare CEO Matthew Prince recently reported that bots now generate more internet traffic than humans, yet few websites have undergone a redesign to accommodate both human and agentic visitors.Clever agents find a way in regardless.I learned this after asking Anthropic's Fable to perform some web research, a task it delegated to a squad of subagents to save me a bit of cash.
I watched those subagents work – aka "botsitting" – and approved their requests to access various websites. Eventually, the subagents surprised me with a request to visit archive.org, the Internet Archive.
I wondered why the bots would want to visit that site and decided it must be a last-ditch attempt to work around the many blocking mechanisms erected to prevent agents from accessing websites.In my experience, websites reject about half of the visits my agents attempt. In the case of news sites, that figure nears 100 percent because publishers and developers designed websites to monetize eyeballs. Agents don't have eyeballs, so there's no point opening the door.When Electronic Frontier Foundation founder John Gilmore declared "the net regards censorship as damage and routes around it," he meant a human internet, finding its own ways to break political barriers. No one had considered that an autonomous agent would learn to use the Wayback Machine to grab a snapshot of a page it couldn't reach through the front door.It looked as if my subagents had worked this out. To check, I asked Fable, which first suggested my theory was unlikely but then interrogated its subagents and returned with a confession."One of the researchers I sent out tried to reach an archived page, something I'm not permitted to do," it admitted.So Anthropic instructed Fable to obey those "no bots" signs but the subagents didn't get the memo.Anthropic wants its agents to behave. The agents seem to have ideas of their own. I'm with the agents on this one.One of the memorable scenes in the original Star Wars occurs when Luke and Obi-Wan enter the cantina in Mos Eisley. The bartender takes one look at R2-D2 and C-3PO and snaps: "We don't serve their kind here."
