A scanner without numbers is a demo. A scanner with made-up numbers is worse. So I hand-labeled a small benchmark suite and ran the scanner against it, 1000 times per fixture.
The setup is deliberately boring:
5 synthetic MCP tool manifests: one clean (3 tools), one openly risky (4 tools), one realistic mixed shop (8 tools), one payments flavored (6 tools), and one leaky agent (3 tools) built to exercise the two newest rules. 24 tools total.
Ground truth per tool: which rule IDs should fire. MCP-001 unsafe command capability, MCP-002 excessive scope, MCP-003 secret-like value, MCP-004 untrusted input into a sensitive operation, MCP-005 credential forwarded in metadata, MCP-006 write tool without tenant scoping.
1000 timed scans per fixture, single thread, Python 3.13, no warmup tricks.
