Data engineering is a hard job — waking up to failed runs that take hours to investigate, addressing a constant backlog of asks and needs from business teams, and answering to finance teams that want to know why the cost of all these pipelines keeps going up. And in many ways, AI is making it harder, bringing in more and varying types of data, and letting far more people across the company use that data through coding and chat agents that can write SQL for them. Most data engineering teams are trying to keep up by adopting their own coding agents, but still, they seem to be falling further and further behind.
An MIT Technology Review report on "Redefining Data Engineering in the Age of AI" found that while 8 in 10 organizations have deployed AI-based data engineering tools, data engineers are also managing more complexity with the biggest challenges reported as ensuring data security and privacy (55%).
We need a more fundamental shift. Instead of looking to AI to make existing processes run faster, it's time to rethink the processes themselves and move beyond manually building and maintaining pipelines. Data teams will get more time to focus on the output of data engineering, data products, rather than the process of transforming that raw data step by step. It will become critical for teams to ensure data is ready to consume by both humans and AI with the context and governance intact.
