As sites attempt to block AI crawlers, is the ‘open web’ closing?

Tai Neilson, a senior lecturer at Macquarie University, explores how data has become a ‘hot commodity’ for companies training AI systems.

When the world wide web went live in the early 1990s, its founders hoped it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking.

The Internet Archive has been recording the history of the internet and making it available to the public through its Wayback Machine since 1996. Now, some of the world’s biggest news outlets are blocking the archive’s access to their pages.

Major publishers – including The Guardian, The New York Times, the Financial Times and USA Today – have confirmed they’re ending the Internet Archive’s access to their content.

While publishers say they support the archive’s preservation mission, they argue unrestricted access creates unintended consequences, exposing journalism to AI crawlers and members of the public trying to skirt their paywalls.