Two months ago a repository turned up in my corner of the internet: 3,358 stars, 744 forks, an MIT license, and a release advertised as 518,400 training samples — 5.5 GB, split across three zip parts. My corner is Chinese astrology software, a domain with almost no machine-readable data, so a corpus that size was either the most useful thing published in years or it was nothing at all. I wanted to know which.
Then I looked at the sample count for one second longer. 518,400 = 60 × 12 × 30 × 12 × 2. Sixty years, twelve months, thirty days, twelve two-hour periods, two genders. That isn't a tally of things somebody observed. That's the size of a nested loop.
Which is a wonderful reason to be curious and a terrible reason to conclude anything. So I went and read the archive — 48 MB of it, over HTTP range requests, never touching the other 5.83 GB. The trip turned out to be more interesting than the destination, and the method is the part that travels well beyond my strange little niche.
So: here's how you read a multi-gigabyte release for the price of a few photos, and then the four things it told me.
First, the trick: reading a 5.5 GB release without downloading it
