I Ran 157 Agent Plans Against a Real LLM. The Problem Wasn't Execution. It Was Planning.

I thought I was building a better planning engine. What I actually built was a machine for showing me how often a decent-looking plan is still wrong in exactly the way that hurts: not obviously wrong, just missing the one dependency or ordering constraint that turns a migration into an incident.

The Failure Starts Before the First Tool Call

Your agent can execute perfectly and still fail, because the plan it was handed was never good.

The whole agent ecosystem is obsessed with execution: tools, memory, orchestration, RAG, function calling, evals. I care about those too. But after building PlannerCritic, I think a lot of teams are optimizing the wrong layer first.

The failures that actually matter often happen before the first tool call.