Every data engineer I talk to has the same story. Somewhere in their company, there is a table with a column full of JSON strings. Maybe it holds event payloads from a mobile app. Maybe it holds sensor readings from a fleet of devices. Maybe it holds the raw output of some third party API that changes its shape every quarter. Whatever the source, that column is both the most valuable and the most painful part of the table.
It is valuable because it holds the raw truth. It is painful because every query that touches it pays a tax. The engine has to read the whole string, parse it, walk the structure, and pull out the one field the query actually wanted. Multiply that by billions of rows and you get slow dashboards, angry analysts, and cloud bills that make finance teams nervous.
Apache Iceberg version 3 of the table format spec introduces the Variant type to fix this problem. Variant gives you the flexibility of JSON with performance that gets close to regular typed columns. The trick that makes this possible is called shredding, and shredding is what I want to explain in this article.
My goal here is not to walk you through the spec line by line. The spec exists and you can read it. My goal is to make the logic of Variant and shredding click for you. I want you to finish this article and think, "oh, of course that is how it works." Once the mental model lands, the spec details, the engine documentation, and the benchmark numbers all become easy to reason about.
