Most "we made it 3x faster" scraper posts miss the actual point. Speed is rarely the constraint....
Most "we made it 3x faster" scraper posts miss the actual point. Speed is rarely the constraint. Replayability is.
If your scraper takes 4 hours to run and a single URL fails halfway through, can you re-run just that URL in 30 seconds? Or do you have to start the whole 4-hour job over?
If the answer is "start over," you don't have a scraper. You have a long-running prayer.
The 3-item checklist
A replayable run looks like this:
Note: This is a cross-post. Canonical version (full long-form) lives on my blog:...
My scraper died at row 12,000 of 50,000, three hours in. The crash itself was cheap. A process gets...
A scraper can pass every check you wrote and still be wrong about the one thing you actually care...
Most Playwright tutorials teach you to scrape a single page. Real scrapers need to scrape thousands....
Best way to find out where your scraper is fragile? Break it. On purpose. In a controlled way, in a...
TL;DR — Migrating a long-running cron-based scraper to an actor architecture does not require a...
