Stop dumping raw message arrays into LLMs and start using structured state with sliding windows.
The Bottleneck in Production
The most common mistake when deploying AI agents is treating chat history as an append-only log. In early prototypes, appending every user turn, tool response, and raw JSON blob directly into the messages array works fine.
In production, this pattern collapses after twenty turns. Token usage scales linearly with conversation depth, driving up API latency and inference costs. Worse, models experience "lost-in-the-middle" degradation, forgetting early constraints or crashing altogether due to token limit errors.
# The naive anti-pattern: unbounded list growth
