Why I Gave Up on Perfect Selectors and Asked GPT to Extract My Data

I’ve been building scrapers for years. CSS selectors, XPath, regex — I’ve written thousands of lines of code just to pull product names and prices from e-commerce sites. Every new site meant a new set of selectors. Sometimes the HTML would change slightly and my entire script would break. It was exhausting.

A few months ago, I needed to monitor prices across 30 different online stores. Each one had a completely different DOM structure. I spent two days writing custom selectors for the first five sites and realized I’d be at this for weeks. There had to be a better way.

What I Tried First (and What Failed)

First, I tried the old reliable: BeautifulSoup with a mix of find_all and select.

from bs4 import BeautifulSoup