I have a scraper that watches ten sites I consider adjacent to my work. It has been running since early July. The output goes into a vector store that nothing currently reads, which is the only reason this story ends without a lawyer in it.
I audited those ten sources this week, against the same bar I use for the security feeds on my site: the licence has to permit commercial reuse in writing. My site sells things. Ambiguous means no.
Two passed. Eight didn't. And the pattern in the failures was not the one I expected.
The two questions are not the same question
Here is Simon Willison's robots.txt, which is about as welcoming as they come:
