I spent a long time believing that a blocked scraper was a headers problem. Wrong User-Agent, missing Accept-Language, something in that family. So I would copy a browser's headers field by field, hit the target again, and get the same 403. Then I would add a proxy. Same 403. Then I would add a delay. Same 403, just slower.
The thing I was missing is that the server had already classified my client before my request headers arrived. It did it in the first packet of the TLS handshake, and no amount of header polishing reaches that packet.
This post is about three things that, once I understood them, reorganized how I build fetchers: the handshake that identifies you, the line between what an HTTP client can present and what it structurally cannot, and the fact that a challenge page usually arrives wearing a 200.
1. The handshake is a fingerprint
Before any HTTP request, your client opens a TLS connection. The first message it sends is the ClientHello. It is plaintext, it is not secret, and it is an announcement: here are the cipher suites I support, in my preferred order; here are the extensions I understand, in this order; here are the elliptic curves I accept; here are the signature algorithms; here is the ALPN list of protocols I am willing to speak afterwards.
