I maintain Village Finder, an open-source mapping project tracking over 78,000 Indian villages. It works by pulling daily raw updates straight from official public administration feeds.
That means my GitHub Actions CI environment hits Indian government web infrastructure every single day. Over time, you accumulate quite a few war stories.
If you're building systems that rely on open data pipelines, these five hard-learned lessons will save you days of baseline debugging.
1. The WAF That Lies to Python-Requests (HTTP 502)
One week, my daily cron job started throwing a wall of HTTP 502 Bad Gateway errors after its eighth automatic retry. Oddly, copying and pasting the exact same endpoint URLs into a desktop browser loaded the JSON data instantly.
