I spent the better part of last month debugging a scraper that had been running fine for almost a year. The symptom was familiar: requests started returning 403s, then CAPTCHAs, then nothing at all. I did what I always did. I added more IPs to the pool. I shortened the rotation interval. I swapped out the proxy provider entirely.
It didn't help.
That's when I realized the thing doing the blocking had changed what it was looking at. And my architecture was built for a problem that no longer existed.
The Old Model: IP Address Was the Signal
For years, the mental model was simple. A website sees a request coming from an IP address. If that IP has made too many requests too fast, or if the IP belongs to a known datacenter range, it gets blocked. The fix was equally simple: rotate IPs. Use residential proxies, spread requests across thousands of addresses, and the blocking system loses track of you.
