8 hours. That's how long a nightly batch job took when I picked it up two weeks ago. The team had already thrown a bigger cluster at it. Twice. It barely moved.
By the end it ran in 47 minutes. Same cluster. Same algorithms. I didn't touch the "smart" part of the pipeline at all.
The whole win was in how the data entered the system. The parser. The first few stages nobody ever profiles because they look boring.
This post is the long version of that fix — written so that if you've only been doing this for a year, you can follow every line. We'll define the jargon as we go, measure before we touch anything, and then walk through the five things that are almost certainly eating your runtime, each with before-and-after code you can run.
First, what is "the layer"? (30 seconds for newcomers)
