Imagine handing an LLM the full text of Anna Karenina and asking what Levin thinks about farming. The model has the information somewhere in its window. Good luck getting a focused answer. The number of tokens you can pass to an LLM in a single call can grow to 200K and beyond, but the problem is not capacity. The problem is that LLMs diverge when there is too much information. Relevance degrades with volume. Past a threshold, adding more context makes answers worse, not better. This is why "just make the window bigger" is not a memory strategy.
AI agents need memory that is selective, persistent, and searchable. Not a transcript. Not a sliding window. A system that extracts what matters, stores it durably, and retrieves only the right pieces when the agent needs them. This is a search problem, and Amazon OpenSearch Service now provides purpose-built APIs for exactly this: agentic memory that gives your agents persistent, semantically searchable recall across conversations.
The context window is not memory
LLMs process text in a fixed-size window. GPT-4o gives you 128K tokens. Claude gives you 200K. That sounds like a lot until you try to use it as a memory system. A single week of customer interactions for one user might run 50K tokens. A month blows past any context limit. And even if the window were infinite, retrieval latency and cost scale linearly with token count. You are paying for every token you stuff in there, and most of it is irrelevant to the current question.
