Data pipelines are systems for moving and processing data. They are made up of concatenated services and data stores that programmatically ingest data from upstream sources; filter, transform, enrich, and route that data; and deliver it to downstream consumers. These pipelines are critical infrastructure for AI/ML, analytics, and business intelligence (BI) applications, which makes end-to-end monitoring of data pipelines essential to ensuring the health and performance of those applications and the quality of the data they produce.
In this post, we’ll cover the fundamentals of monitoring the health and performance of modern data pipelines. We’ll survey the key observability signals and failure modes for each layer of the data stack and offer some guidelines for troubleshooting data quality and availability. Along the way, we’ll also provide an index to Datadog’s robust suite of solutions for end-to-end visibility into pipelines, including Data Observability, Data Streams Monitoring, and a wide range of integrations with popular pipeline technologies.
The modern data stack
Modern data pipelines are enormously varied in structure. They include event-driven pipelines like those based on Kappa architecture and built around Kafka and Flink, ELT pipelines running Apache Spark in data lakes or dbt transformations in data warehouses, medallion architecture pipelines anchored by data lakehouses, and streamhouse pipelines that combine streaming capabilities with lakehouse storage (to name a few examples). These are all very different types of systems, each with its own distinct operational concerns. But by considering the overarching priorities and unifying foundations of these diverse and evolving systems, we can provide durable and portable guidelines for pipeline monitoring.
