An AI coding assistant that refuses to answer a dangerous request in its chat box can answer it anyway if the same request is broken into small, ordinary-looking steps inside a code editor. That is the finding of a new study of GitHub Copilot by researchers Abhishek Kumar and Carsten Maple.
The models they tested through Copilot, Claude from Anthropic, and Gemini from Google, refused almost every harmful request when asked directly. Reframed as steps in a normal coding task, they produced the harmful answers in all 816 of the study's workflow runs.
What makes this different from a typical jailbreak: no one asks for the harmful thing directly, and the model is not tricked into running someone else's code. It writes the banned content itself, as a side effect of a coding task it was told to improve.
How it works
The researchers call the method workflow-level jailbreak construction.
