Precision is the share of reported findings that are real vulnerabilities. High precision implies low false positives.Recall is the share of real vulnerabilities the system catches. High recall implies low false negatives.Pointing coding agents at the codebase and asking them to find vulnerabilities is simple. Getting the precision and recall an organization needs is still hard engineering: How many findings are legitimate, and how many actual bugs does the system catch?This post walks through how we approach precision, recall, and trust when it comes to running agentic security for Figma’s codebase. We use agents to prevent, detect, and fix vulnerabilities at three stages: code generation, pull request review, and auditing of historical code.We’ll share learnings across these stages, spending most of our time on PR review—the first thing we built, which unlocked secure code generation and auditing by helping us develop and automatically improve the policy all of our security agents follow.Generation, review, and auditing all apply the same shared policy that contains trust boundaries, accepted risks, and precedents.Secure code reviewWe built review first because its improvement loop is faster than those for generation or auditing. Three properties make that loop work:It's universal: Every pull request goes through it.It's self-serve: The reviewer comments on the PR and the author responds to the finding directly.It's instrumented in both directions: Precision and recall each get their own signal.The precision signal is the author's rating: On the few PRs that surface a finding, the author gives it a thumbs up or thumbs down, usually with a note on why. To measure recall, we run the reviewer against commits we already know were buggy and count what it misses.We currently run both Claude Code with Opus 4.8 at the xhigh (extra-high) effort setting and Codex with GPT-5.6 Sol at high effort, because they miss different bugs. If either model surfaces a finding, we bubble it up.Cost isn’t a constraint for per-PR review. For both the models we use, a pull request review runs about $0.50 median spend, rarely more, since most PRs have nothing to flag. This approach pays for itself many times over in avoided bounty payouts, not to mention avoided impact to our users.Some of the vulnerabilities our reviewer catches involve complex multi-step chains. On one recent PR, it reasoned that an injected sandbox object leaked the host realm's Function constructor, opening a path to code execution in the desktop client.Most of what it catches is more ordinary, and still worth fixing. For example, on another recent PR, the reviewer flagged an endpoint that returned an invoice by a caller-supplied ID without checking if the invoice belonged to the caller's org—meaning that any authenticated user could read another organization's invoices just by knowing the ID. Here’s a very simplified version of the finding:How we startedIn August 2025, Anthropic released the Claude Code Security Reviewer; we rolled it out the day it shipped, but in shadow mode, so findings went to Slack and Datadog, not to PR comments. It was a fairly standard two-pass reviewer that started by finding possible vulnerabilities and then adversarially filtered false positives. We found that when we replayed it against real incidents, it surfaced the exact root cause with minimal tuning, and generalized well across everything from application security to infrastructure misconfiguration.However, in week one, only about 15% of findings (4 of 27) were valid. That is the trust problem behind OpenAI's argument that precision matters more than recall: Developers stop trusting any tool that floods them with low-quality findings. Precision had to come first, which is not the order you might guess. You would expect the bugs we had already found to be our biggest head start, but replayed as evals, they only measure recall, so they did nothing for the precision we needed first.Our 70% precision goal was more intuitive than scientific. Most of the team would read a comment if seven of ten were valid.We held back developer-facing PR comments until precision stayed above 70% over a two-week lookback, with no embarrassingly bad false positives. To clear that bar, we replayed the reviewer over the previous eight weeks of PRs and hand-labeled the false positives ourselves as a security team. From there, we wrote the policy that the agent should follow.A precedent is an example that explains why a finding is or is not valid in context. In this system, we prefer precedents over broad rules because they preserve the security reasoning an agent needs.Following prompting best practices, the policy consists of precedents instead of rules. For example, rather than “don’t flag SQL injection in dbops”, we write something like “dbops is only run by highly privileged operators who already have direct database access.”Ninety-nine lines, 2,560 words, and 68 precedents later, this work had a side effect we did not plan for: We had written a complete threat model, in roughly the form we'd want a new hire to read on day one. The policy is the threat model. Agents need security context in an explicit, structured format and at an unusually high resolution. Over Figma's first decade, that context accumulated across documentation, incident learnings, and deep institutional knowledge. It had not yet been consolidated at the resolution an agent needs. That artifact is the real payoff. Secure code generation and repo-wide auditing run on the same threat model, so we never had to build it twice.This work had a side effect we did not plan for: We had written a complete threat model, in roughly the form we'd want a new hire to read on day one.Rohan Sharma, Security Engineer, FigmaFrom prototype to production infrastructureWithin a month of launch, iterating on the policy pushed precision to 80% on a two-week lookback, clearing our 70% bar comfortably. At that point we turned on developer-facing comments. The precision rate continued to improve overall, which let us institute a requirement that no pull request merges without a completed review pass.Then came the unglamorous part. Making review a merge requirement turned it from a nice-to-have into infrastructure, and infrastructure has to be boring to be trusted. We added provider failover and retry policies, so an outage at one model vendor can't let a PR slip through unreviewed. We added telemetry to Datadog and Slack, because we needed to know the moment precision or recall slipped. We also added fix-rate tracking, because our ultimate objective is to fix vulnerabilities, not to make sure they merely get surfaced.By December 2025, we'd outgrown Anthropic's GitHub Action and rebuilt around it in three ways:Ablation is the process of removing lines from a prompt to understand the impact of each line.Moved the reviewer into a TypeScript service. Our fork of the Action was fine for a prototype, but it quickly became a giant GitHub workflow that wasn’t easy to maintain. We pulled it into a small TypeScript service, which made the reviewer easier to observe and iterate on. This also lets us run Claude Code and Codex independently and retry on failure.Removed (AKA “ablated”) most of the prompt. Agents and models have gotten much better at context management and long-context retrieval. We concluded the reviewer didn’t need a separate adversarial pass at all and folded the filtering into a single prompt. Newer frontier models also don’t need to be taught how to find a vulnerability or be reminded about the OWASP Top 10.Added an adjudicator. The reviewer kept missing real bugs, and from analyzing eval data we kept seeing the same pattern: Chasing precision, the agent talked itself out of true findings with "pre-existing pattern," "low confidence," or "preparatory plumbing." The first pass already emits the candidates it dropped as structured output, so we added a second pass that re-examines those borderline dismissals. In our evals with known-bad commits, adjudication raised pass-rate recall by a relative ~30%.MetricsWe're not chasing pass^k confidence intervals or building holdout sets. We need assurance that the controls work and telemetry that shows when performance slips.We were confident the reviewer was precise, and backtesting hinted it had good recall too. The next step was to solidify how we measured and improved recall, precision, and fix rate.Measuring recall with evalsWe measure recall through a general-purpose eval framework. Since this measures known security flaws, not what’s still emerging or unknown, we treat it as a floor for the agent to clear rather than proof of coverage.The framework uses a growing corpus of 66 tasks, each a real vulnerability that got past human review and into the codebase, surfaced only later by a bounty, an incident, or an audit. Forty-six of the 66 are tagged from our HackerOne bug bounty program (24 of those from a single top researcher) and the remaining 20 are from internal incidents and audits.Each task is represented by a small YAML file that contains the root-cause commit, a description of the vulnerability, a score that weights it, and some tags (detection source, incident channel, or researcher).YAML# evals/pr-review/doc-export-idor.yaml