The Planner Made the Same 3 Mistakes Every Time. A Bigger Model Didn't Fix It.

This is article 3 in a series about building PlannerCritic, an open-source engine where one LLM writes a plan and a second LLM reviews it. Article 1 covers the 157-goal field test. Article 2 is about the critic severity bug. This one is about the most uncomfortable thing the field test revealed: the planner has a structural problem that no model upgrade fixes.

132 concrete blockers across 63 strict goals. Three defect families. I tried gpt-4o. Same pattern. The fix is deterministic validation, not more parameters.

The Pattern

By the 10th strict goal, I noticed it. By the 50th, I could predict the failure before the critic printed it. By the 100th, I stopped being surprised and started being annoyed.

The planner kept making the same three mistakes. Not occasionally. Not randomly. Every single strict goal that failed did so because of one of three defect families.