Stop letting the prompt be your state machine

Stop letting the prompt be your state machine

You shipped an LLM feature six months ago. Now the same user input produces wildly different outputs depending on... nothing you can point to. Something in the sampling? The time the context filled up and a chunk got dropped? Nobody knows. This is what happens when the prompt becomes your runtime.

The trap: the prompt as an accidental runtime

Here is what the trap looks like in TypeScript:

async function handleUserRequest(input: string): Promise<string> {