Taking AUTO CDC to the next level: Solving the hardest real-world use cases

From bitemporal compliance to partial record updates: delivering robust, audit-ready change data capture without custom code

by Josh Seidel, Shanelle Roman and Sudhanva Huruli

Change data capture is one of the most common things data engineers build on Spark, and one of the most tedious to get right by hand. In our previous post, Stop hand-coding change data capture pipelines, we introduced how AUTO CDC in Apache™ Spark Declarative Pipelines (SDP) automates SCD Type 1, SCD Type 2, and Snapshot CDC by replacing hundreds of lines of fragile MERGE logic with a few simple declarations.

As pipeline requirements evolve, engineers run into situations that those standard CDC patterns struggle to solve:

Today, we’re taking AUTO CDC to the next level to solve these exact real-world challenges, and expanding these capabilities into open-source Apache Spark 4.2.