I built an AI that pentests my AI — and forced it to prove every exploit
Point an LLM at your own system and tell it to "find security vulnerabilities" and you'll get a page of confident, well-formatted, mostly useless prose. "This endpoint may be vulnerable to prompt injection." "The tenant filter could potentially be bypassed." Could. May. Potentially. You can't tell a real exploit from a hallucinated one, so you either chase every claim or trust none of them. Either way the report is worth nothing — and worse, it feels like security work while being none.
That unfalsifiability is the whole problem with AI-driven pentesting, and it's the thing I set out to kill when I built agent-redteam — a local, Claude-orchestrated adversarial harness that attacks a real production copilot over a regulated document store (a LangGraph agent) and reports only exploits it can prove.
The frame for the threat model came from Anthropic's "Zero Trust for AI Agents", which names a handful of agent threat categories and pitches "defensive operations at attacker speed." Good article. But reading it, the useful move wasn't to admire the taxonomy — it was to invert it. That list of threats isn't a threat model to nod along to. It's a test plan you can automate.
