OpenAI's audit reveals 30% of tasks in SWE-Bench Pro are flawed, prompting the company to withdraw its recommendation for replacing SWE-bench Verified with this benchmark.
Nearly 30% of SWE-Bench Pro tasks are flawed
Jul 09, 2026
01:04 pm
What's the storyOpenAI has raised concerns over SWE-Bench Pro, a popular coding benchmark.
The company conducted an audit of the test and found that nearly 30% of its tasks are flawed in ways that misrepresent what an AI model can actually do.
OpenAI just found ~30% of SWE-Bench Pro is broken — and retracted their own recommendation
OpenAI finds roughly 30 percent of popular AI coding test is broken
Most Evals Measure the Wrong Thing
Artificial Analysis launches EnterpriseOps-Gym-AA to benchmark AI agents in real enterprise systems
I Spent a Week Fixing the Wrong Skill (And Other Lessons from Evaluating an AI PR Reviewer)
OpenAI pulled the plug on SWE-bench Verified earlier this year after finding contamination and design...
OpenAI reviewed SWE-Bench Pro, a widely used test for measuring AI models' programming skills, and found roughly 30 percent of…
Turn a coding-benchmark audit into a reproducible task-state model, sensitivity analysis, and acceptance rule.
‘We’ve identified multiple loopholes with SWE-bench Verified,’ the manager at Meta Platforms’ AI research lab Fair says.
A visual explanation for OpenAI's new science for coding evaluations and benchmarks.
To fix the way we test and measure models, AI is learning tricks from social science.
