I recently built a multi-agent customer support system on Azure AI Foundry and NVIDIA NIM. First time doing anything like this. Made four predictions upfront about what would happen. Three of them were wrong.
Here is what I actually learned.
1. "Tokens" is not a unit of cost
It is a unit of work. The price per unit of work varies by 5-10x depending on which model did the work. I was tracking total token count across both the small 9B model and the large 49B model as if they cost the same. They do not. Total tokens went up in the optimized version. Cost in dollars probably went down. I was measuring the wrong thing the whole time.
2. A verbatim hash cache on natural language traffic deflects ~0% of queries
