A reader who first used the web in 2020 does not remember what a search results page looked like in 2003. It is worth describing in some detail, because the gap between that page and the one you used this morning is the subject of this piece, and the size of the gap is what makes the story interesting.
In November 2003, on Google, Altavista, AskJeeves, or any of the smaller engines whose names have not survived, a top-ten search results page for a moderately commercial query — cheap car insurance, prescription drugs, download windows xp — typically contained one or two pages a human had written, sandwiched between eight or nine pages a small script had assembled. The script pages were called doorway pages. Each one was a thin shell of keywords with a redirect to a target site. The target site sold something or showed ads. The economics of the operation were that a single person could produce ten thousand doorway pages a week for under a hundred dollars and rank some non-trivial fraction of them on at least one query. The economics worked. The results page was the consequence.
This was the visible web for several years. The story of the last twenty-three years is the story of search engines, and one search engine in particular, slowly cleaning that page up. It is not a story with a happy ending. It is a story with a long middle, several decisive battles, and an open final chapter that began in 2023 and has not yet closed.
