OpenAI just found ~30% of SWE-Bench Pro is broken — and retracted their own recommendation

OpenAI finds roughly 30 percent of popular AI coding test is broken

OpenAI reviewed SWE-Bench Pro, a widely used test for measuring AI models' programming skills, and found roughly 30 percent of its tasks are broken. The company is pulling its earlier endorsement of the benchmark.