This is article 2 in a series about building PlannerCritic, an open-source engine where one LLM...
This is article 2 in a series about building PlannerCritic, an open-source engine where one LLM writes a plan and a second LLM reviews it. Article 1 is here — it covers the 157-goal field test and what it found. This one is about a specific bug that taught me something about LLM judgment.
The critic was supposed to catch unsafe plans. Instead it blocked plans for being incomplete. The fix was a frozenset.
What Happened
I gave the critic this system prompt:
You are an adversarial plan reviewer.
This is article 3 in a series about building PlannerCritic, an open-source engine where one LLM...
This is article 5 in a series about building PlannerCritic, an open-source engine where one LLM...
Why I used three different critic roles instead of one (and what the eval taught me) I...
Large language models are surprisingly optimistic reviewers. Ask an LLM to review an implementation...
10 adversarial scenarios, 64 assertions, 3-tier evaluation pyramid. Llama, Qwen, GPT-OSS — none scored above 63%. Here's what…
When I shipped v0.2.1 of PlannerCritic, I thought the hard part was over. The engine had survived a...
