Most scrapers have one retry policy: catch the exception, sleep, try again, give up after N attempts. It's the default in every tutorial, it's what tenacity gives you in three lines, and on a healthy target it's fine.
It stops being fine the moment the failures stop being random. A rate limit, a hard block, a TLS reset and a slow origin all arrive as "the request failed," and treating them identically means you spend your entire run budget hammering something that was never going to open — while the one failure that would have succeeded on retry gets three attempts and then gets dropped.
Here's the taxonomy I actually use, and the part that took the longest to learn: the counter matters more than the backoff.
The four failures, and what each one means
429 with a Retry-After header. The friendliest failure you will ever get. The server is telling you exactly what it wants. Honour it — sleep the stated duration, then continue on the same connection and the same identity. Do not rotate anything. Rotating here is how a temporary throttle turns into a fingerprinted pattern.
