OpenAI reviewed SWE-Bench Pro, a widely used test for measuring AI models' programming skills, and found roughly 30 percent of its tasks are broken. The company is pulling its earlier endorsement of the benchmark.
Results from tests like these feed into decisions about whether and how to release a model, including safety assessments under OpenAI's Preparedness Framework. When a test contains errors, it can paint a misleading picture of what an AI can actually do.
To run the review, OpenAI first deployed an automated screening tool that flagged 286 suspicious tasks. AI agents built on the Codex model then examined each case in detail before a human researcher made the final call. That process labeled 200 tasks (27.4 percent) as flawed. In a parallel review, five experienced software developers evaluated the same cases and flagged even more, 249 tasks (34.1 percent). The human reviewers were stricter than the AI agents, though both sides agreed in 74 percent of cases.
A single whitespace character can mean pass or fail
OpenAI breaks the problems into four categories. Some tests are too strict, rejecting solutions that actually work. Others are too vague, expecting the AI to meet requirements buried in hidden test cases. Some tests are too shallow, letting incomplete solutions pass. And some task descriptions simply point in the wrong direction. One example from the OpenLibrary project: the task description called for a single space, but the hidden test expected two. An AI that correctly followed the instructions would fail.
