The LLM was better at building a solver than playing the game

I started this project because an LLM annoyed me.

I gave a very strong model 322, a small Dota 2 drafting game. The choices looked like the kind of work a computer should enjoy: repeated packs of players and heroes, visible ratings, familiarity scores, chemistry, rerolls and a simulated tournament at the end.

I was disappointed by how well the LLM did. I am not a Dota expert, and I had only started watching it occasionally again during the previous six months or year. I still seemed to be doing better.

The interesting engineering question was not how to write a longer prompt. It was how to replace the card-by-card language-model judgement with a deterministic policy, then test that policy without confusing improvement with luck.

A stochastic benchmark needs shared randomness