Hi Everyone!
We had this classic pain point on our project: constantly chewing through massive JSON arrays. Catalogs, analytics dumps, ML datasets — files ranging from a couple of hundred megabytes to tens of gigabytes.
The task was stupidly simple: split a giant JSON array into individual elements so we could chunk them or throw them into parallel processing. No data transformation, no querying by keys. We literally just needed to find where each chunk starts and ends.
Naturally, we started with the classic approach: json.Unmarshal -> slice -> json.Marshal. On a 10GB file, memory consumption went to the moon 🚀. We ended up spending more time fighting the Go garbage collector (GC) than doing actual work.
And then it clicked: to just move the data around, we don't need to understand what's inside it. We just need to find the boundaries.
