When we hand off an LLM-powered feature to a client, they are putting it in front of their users. Some of those users will try to break it — not always maliciously, but because users probe things. Some of them will be malicious. Either way, the question we ask before every handoff is: what happens when someone sends this system something it wasn't designed to handle?
This post is our pre-shipment checklist for LLM integrations. It covers prompt injection, output filtering, data exposure, and a few failure modes we have seen in the wild that are easy to miss.
The threat model for LLM features
LLM features are different from conventional features in one important way: the attack surface includes natural language. An attacker does not need to know your API schema — they just need to craft a sentence that changes what your system does.
The most common classes of issues we see:
