We run an IP law practice. At some point our internal tooling needed to search the Indian Trade Marks Registry programmatically, so we scraped it — 3.3M+ records, going back to Trade Marks Journal issue 1703. Someone building a brand-clearance tool asked if that data was available as an API. It wasn't, so we shipped one.
This isn't a "look at our product" post. It's what the underlying data source actually does to you when you try to read it at scale, plus the API we ended up with.
The registry does not want to be scraped
A few specific things broke us, in no particular order:
Inconsistent filenames across journal issues. The Trade Marks Journal (currently in the 2200s) uses two different filename conventions depending roughly on when the issue was published — no documented boundary, no changelog. Guess wrong and you get a 500 with an HTML error page, not a 404. If you're not checking content-type before parsing, that HTML silently becomes garbage input to your parser.
