My app generates personalized readings for BaZi — Chinese "Four Pillars" birth charts. Every reading is an LLM call, every call costs money, and the domain is full of trap terminology that models love to botch. So before launch I benchmarked every candidate model on my actual workload, and then built the routing layer around what the benchmark found.
The results generalize to any "LLM in a niche domain" app, so here they are — including the part where the most expensive model lost to one costing 5.8× less.
What "good" means in a niche domain
Generic benchmarks were useless to me. My acceptance criteria were:
Domain accuracy: 甲 is Yang Wood. A model that renders it "Yin Wood" in English output is not 5% wrong, it's categorically wrong — the way a compiler that flips one bit is wrong.
