The Day I Realized "It Ran Successfully" Means Nothing in Databricks Production

Here's a scenario that plays out at every company that has ever adopted Databricks:

A senior engineer ships a new pipeline to production. It's a MERGE job — reads from a silver Delta table, applies some business logic, upserts into a gold table that feeds the executive dashboard. They tested it in dev on a 40 GB sample. Ran in four minutes. Clean output. Passed code review.

What they didn't know: the gold table in production is 3.8 TB and has no Z-ordering on the merge key. No partition pruning either. So every time that job runs, Databricks scans the entire table — all 3.8 TB — to find the rows that need updating.

The job still succeeds. Exit code 0. Green checkmark in the Workflows UI. The output is correct — just catastrophically expensive to produce. It runs every 30 minutes. Nobody sets a cost alert.

That pipeline runs for eleven days before someone pulls the monthly DBU report. $2,300 in compute. For a job that should have cost about $40.