TL;DR
Anthropic's crawler now retrieves 60,000 pages per visitor sent back — up from 6,000:1 six months ago. The website as distribution channel is being extracted, not visited.
Three consumer types have incompatible requirements: humans want narrative, LLMs want dense signal, agents want machine-readable endpoints. Almost nobody is building for the third.
llms.txt (1,000+ sites) is the right model but static — it describes content; it doesn't serve it.
The emerging layer is MCP endpoints: dynamic, agent-queryable knowledge interfaces where monetization is context access, not page views.
