How CLA built a Databricks-native solution for long-running tasks, observability and cost attribution
by Li Yu, Michelle JanneyCoyle, Jon Cormack, Yarri Bryn, Alec Sorensen and Darshana Nair
Traditionally, auditing is a tedious process that often requires detailed document review and information extraction. To accelerate this process, CLA (CliftonLarsonAllen LLP), a leading professional services firm with a growing global presence, worked with Databricks Forward Deployed Engineering team to build and productionize an agentic auditing solution. Jointly, we developed a document processing application that reduces extraction time from hours to minutes with no compromise in quality. The application is built entirely on Databricks, using Lakebase Postgres, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes. In this blog, we focus on one key component of that system, the Lakebase-powered orchestration layer.
The orchestration layer is responsible for coordinating long-running tasks, managing retries, attributing cost, and providing real-time visibility. Lakebase and Databricks Apps, we eliminate the need for separate infrastructure for queueing, orchestration, and observability.
