Booking.com does not want bots, and it shows. Every plain HTTP request, any client, any proxy, gets absorbed by AWS WAF with a 202 that never resolves into content. There is no getting around a real browser here, which makes this the most expensive scrape in my portfolio. What kept the project alive was discovering that the two hard parts, search and data extraction, both have quiet side doors.
Don't touch the search box
My first version drove the UI like a user: open the homepage, type the city, click search. It was a disaster. The autocomplete is timing-sensitive, the results panel loads lazily, and about a third of runs ended with zero hotels for no visible reason. Debugging flaky UI automation is the worst kind of debugging because every failure looks different.
The side door: Booking maintains SEO landing pages per city with stable URLs:
https://www.booking.com/city/{countryCode}/{citySlug}.{lang}.html
