If you’ve ever tried to feed a massive document into an AI model and watched it forget what was on page two by the time it reached page fifty, you’re familiar with the long-context problem. Amazon AGI researchers think they’ve found a way to make that experience significantly less painful, and the key insight is surprisingly elegant: the way you manage a model’s memory during inference should also shape how you train it in the first place.

The paper, titled “Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models,” was submitted to arXiv by a team led by Francesco Maria Molfese, Momchil Hardalov, Rexhina Blloshmi, Bill Byrne, and Adrià de Gispert. Its central finding is that a reinforcement learning technique called Group Relative Policy Optimization, or GRPO, can dramatically improve a language model’s ability to find and use information buried deep within long contexts.

What KV-cache compression actually means for model performance

To understand why this matters, a quick primer on KV-cache. When a large language model processes text, it stores key-value pairs representing the tokens it has already seen. For short texts, this works fine. For documents stretching to 32,000 tokens or beyond, that cache becomes enormous, expensive, and slow.