Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fails. The answer used to default to Airflow because there wasn't a real alternative. Now there are three reasonable defaults, and they optimize for different things. Picking wrong doesn't break anything on day one — it shows up eighteen months later as either an operations team drowning in scheduler maintenance or an engineering team fighting a platform that won't do what they need it to.
Here's the actual tradeoff, not the vendor pitch version.
Databricks Workflows: the path of least resistance, if you're all-in on Databricks
Databricks Workflows is the orchestrator built into the platform. Jobs, clusters, Unity Catalog permissions, and Workflows all share the same control plane, which means you're not maintaining a separate scheduler, not managing a second set of credentials, and not debugging why an external system can't see a table that Unity Catalog says it can. Task dependencies, retries, cluster reuse across tasks, and job-level alerting all come for free.
The cost is exactly what you'd expect from a platform-native tool: it orchestrates Databricks well and everything else poorly. There's no first-class way to trigger a task in your orchestration DAG that waits on a Salesforce export, calls an internal API, or coordinates a dbt run against a warehouse that isn't Databricks SQL. You can bolt these in with webhooks and external scripts, but you're fighting the tool rather than using it. Workflows also doesn't give you the asset-lineage or testing story that Dagster does — it schedules tasks, not data assets.
