This is part four of a series about pointing an append-only audit log at things that count tokens. Part one found that a missing model line was half my agent overspend. Part two found two accounting bugs in splitrail. Part three found a 2.35× over-count in a 30k-star repo that the vendor's own documentation had warned about.
I promised this post at the end of part two and pushed it at the start of part three. Here it is.
The first three parts asked whether tools counted correctly. This one assumes the counting is right and asks the next question: given trustworthy numbers, do they say what I think they say about my own behaviour?
I wrote my routing policy down as a file. Then I measured which model actually ran, per decision, and priced the difference.
425 decisions. 96 deviations, 22.6%, $1,248.13.
