For a decade, the file format layer was the most settled real estate in data. Apache Parquet held the analytical world, ORC held the Hive legacy estates, and the interesting arguments all happened in the layers above. Then, in the span of about three years, the bottom of the stack became the most intellectually active corner of the industry: a research wave produced BtrBlocks, FastLanes, ALP, and FSST, startups building AI infrastructure shipped Lance and Vortex, Meta open-sourced Nimble from its ML platform, and academic groups started publishing formats with names like F3, literally File Format for the Future.
The humble file format is having its renaissance, and the causes are worth stating precisely, because they explain everything about the new entrants. Cause one: AI workloads broke Parquet's assumptions. The 2013 design assumed batch scans over modest-width tables of numbers, strings, and dates. The 2026 workload includes point lookups into billion-row vector datasets, training pipelines shredding wide feature tables at GPU speed, and multimodal blobs sitting next to structured columns. Cause two: hardware evolved past the design. NVMe made storage fast enough that decompression became the bottleneck, SIMD widths grew, GPUs became first-class data consumers, and the heavyweight general-purpose codecs Parquet leaned on stopped being the right trade.
