When we shipped the first version of AI-generated replies for HelperX, each reply cost us about $0.011 in API spend. That sounds tiny until you multiply by 30 replies per slot per day times 200 active slots: roughly $66 per day, or ~$2,000 per month. Not catastrophic, but enough to eat into margins on the smaller plans.
A year later, we're spending $0.0009 per reply — a 12x reduction. Same model providers, similar reply quality, same throughput. The savings came from four optimization layers stacked on top of each other.
This is exactly what each layer does, the order we applied them, and the cost reduction each one produced.
The starting point
The naive implementation looked like this:
