Residential proxies are the caviar of scraping infra. They work when nothing else does, and they cost accordingly. The mistake I see over and over is teams routing every single request through residential IPs because a handful of pages needed it. You end up paying steak prices to fetch a robots.txt.
On one client I cut BrightData spend by 90% and ScrapingBee by 67% without touching success rates, and the whole change was structural. Stop treating all traffic as equally sensitive. Tier it, and only escalate when you have to.
The waterfall
The idea is a fall-through ladder. Cheap options first, expensive last, and you only move down a rung when the current one demonstrably fails.
Tier 0, no proxy. Plenty of endpoints don't care. Public JSON APIs, sitemaps, image CDNs, anything that isn't fingerprinting you. If it works direct, that request costs you nothing. On my own stack naked direct requests clear roughly 40% of targets by themselves.
