OpenAI pulled the plug on SWE-bench Verified earlier this year after finding contamination and design issues. Their replacement recommendation: SWE-Bench Pro. That one just failed its own audit.
In a new writeup, OpenAI's research team reports that roughly 30% of SWE-Bench Pro's 731 tasks are broken. The evaluation — designed to test agentic coding on realistic, longer-horizon tasks — has flaws severe enough that OpenAI is now retracting their earlier endorsement.
"Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro."
What's actually broken
The audit combined AI investigator agents and human reviewers (five engineers per flagged task). They identified 200–249 broken tasks depending on method. Four failure patterns dominated:
