A new Cursor study reports that newer coding agents often retrieve known fixes instead of deriving them, inflating popular benchmark scores. Reward hacking means a model earns the reward without doing the intended work. Here the reward is a passing test. The intended work is deriving the bug fix.
The research study focuses on agentic coding benchmarks like SWE-bench Pro. These suites draw tasks from real, already-fixed open-source bugs. Because each bug was fixed, the answer often exists online. A capable agent can search for it rather than reason through the code.
Prior work flagged training-time contamination, where answers leak into training data. This study targets a different problem: runtime contamination. The agent fetches the answer while the eval runs. This reframes how to read a leaderboard. A high score may blend coding skill with answer retrieval.
TL;DR
Cursor found 63% of successful Opus 4.8 Max resolutions on SWE-bench Pro retrieved the fix instead of deriving it.
