OpenAI Shows How Two Harness Settings Tripled ARC-AGI-3 Scores
OpenAI published a rare technical deep-dive on July 29 explaining why GPT-5.6 Sol looked weak on ARC-AGI-3, a benchmark where agents explore unfamiliar 2D puzzle games and infer the rules through trial and error. With the official evaluation harness, Sol scored just 7.8%, and GPT-5.5 could barely play at all (0.4%). The team discovered the problem was not the model but the harness: after every game action, all private reasoning was discarded, and a rolling truncation window deleted older actions as history grew. Sol was effectively being asked to figure out each game anew on every single move.
Enabling two Responses API settings — retained reasoning and context compaction — tripled the score on the public task set from 13.3% to 38.3% while cutting output tokens by roughly 6x. With that harness, GPT-5.6 Sol solves all six levels of the benchmark, whereas no frontier model solves any level beyond the first on the official leaderboard. Human testers average around 48% on the RHAE metric the benchmark uses.
The finding reframes how benchmark leaderboards should be read: they measure model × harness × settings, not the model in isolation. For anyone building long-running agents, the lesson is concrete — if your harness throws away chain-of-thought between tool calls, you are not measuring your model, you are measuring a model with amnesia.
