We had a healthcare claims platform stuck at 45% recall on medical coding. Not 45% on a benchmark. 45% recall on CPT and ICD code assignment, in a system whose job is getting those codes right before a claim goes out. Get it wrong, and a claim gets denied. A billing team spends an afternoon on the phone with a payer instead of doing their actual job.
This post is about the architectural decision that moved CPT recall from 45% to 92%, and why no amount of prompt engineering or fine-tuning could have produced the same result on its own.
Why fine-tuning stopped working
When a number is low, the obvious move is a bigger model. We made that move too, and it wasn't a bad instinct, at least at first.
We spent real weeks on it: sharper prompts, more examples, better retrieval so the right context sat in front of the model when it made a call. Recall climbed out of the 40s into the high 50s, low 60s. Genuine progress.
