When Databricks claimed to have cracked an age-old database problem, it came with a clear marketing message: "One data, zero compromises, zero copies." Inevitably, that led engineers to search for clarity. After all, the company claimed to have unified OLTP and OLAP with "no data duplication."Databricks, which was founded around the open source unified analytics engine Apache Spark, called its invention LTAP, which stands for lake transactional/analytical processing. It works with Reyden – a new compute engine – and Lakebase, its serverless PostgreSQL on open object storage.
Databricks is attempting to address a fundamental database challenge. OLTP (online transactional processing) performs small, row-oriented reads and frequent writes, while OLAP (online analytical processing) performs large, column-oriented reads and batch writes. Down to the physical level, it is challenging to get the two to coexist in a single system. The issue is seen as more pressing now as the database market chases workloads created by the booming deployment of AI agents, both in software development and business applications.
What did Databricks claim? The publicity material said that rather than forcing both OLTP and OLAP workloads into one engine or concealing the pipeline, it unifies data at the storage layer, thereby unifying transactions, analytics, streaming, and operational data on a single copy of storage in the data lakehouse, a concept Databricks created to describe the marriage of data lakes and data warehouses. Does that mean there are "zero copies" of the data, as claimed in several promotional LinkedIn pieces and a Forbes CEO interview? Well, not quite. The transactional side of LTAP is based on Databricks' first fully managed PostgreSQL database, Lakebase, which in turn is based on technology from Neon, which Databricks bought last year to provide copy-on-write branching and autoscaling serverless compute.In his search for clarity, one data engineer in financial services posted that LTAP proposes that the current PostgreSQL data stays in the pageserver format as local storage then is propagated to object storeage for long-term durability in the Parquet file format, where it can be queried in a columnar format. PostgreSQL/Lakebase can retrieve data from the object store and reconvert the Parquet data to a pageserver if it needs data from cold storage. In this way, Databricks has "unified" the OLTP storage and OLAP storage."Two copies of data, not one," quipped one commenter from a Databricks rival. Slides made available at a PostgreSQL conference in May make the link clear. Under the header "Analytics directly on OLTP data," Databricks engineers Hristo Stoyanov and Jonathan Katz said that pageserver provides storage while the Spark analytics executor pulls layer files containing full page images from the image layers in object storage. On a private messaging community seen by The Register, one Databricks engineer responded to the question about whether there was one copy of the data or two copies in object storage and pageservers respectively. Technically two, they responded, since pageservers act as a cache or materialization layer in the Neon architecture. PostgreSQL reads from pageservers, while the analytics engine reads PostgreSQL pages from object storage (Apache Parquet or Iceberg table format) and pageservers. Databricks is far from alone in trying to crack this nut. Unifying OLTP and OLAP has been tried before, and solved, according to some companies. For example, in 2014, SingleStore began working on an in-memory row store and an on-disk column store with tiered storage, "meaning transactions hit memory first and then they roll off to disk storage," allowing analytics and transactions on a single system. It launched a cloud database service (on AWS, Azure, or GCP) in 2020, which "automatically manages data across a three-tiered storage architecture comprised of memory, local cache, and storage." It moves data "seamlessly" between memory, persistent cache, and object storage without the user being aware, the company says.
