I Benchmarked 6 Prompting Strategies on Two Models. The Winner Changes Depending on Which Model You Ask.

Over two weeks, I built a small evaluation harness to test whether popular prompting techniques — few-shot examples, Chain-of-Thought, self-consistency voting, Tree-of-Thought — actually improve accuracy, and at what cost. I ran everything on 20 fixed GSM8K math word problems, first against a free local model (Mistral-7B via Ollama), then again against GPT-4o-mini via the OpenAI API, using identical questions and prompts.

Full code and raw results: github.com/Mohsin2686/prompt-bench

The headline finding: no technique is universally "best"

On Mistral-7B, self-consistency (sampling 5 answers and majority-voting) was the top strategy at 45% accuracy. On GPT-4o-mini, self-consistency was tied for the worst strategy at 75% — actually underperforming a plain zero-shot prompt. Meanwhile, few-shot prompting was Mistral's worst technique (35%, actively hurt by showing three worked examples) but GPT-4o-mini's best (80%, tied with Chain-of-Thought).

Strategy