Current end-to-end systems use a language model as their decoder, so this buffer grows with every new line of text. That drives up memory use and steadily slows generation. In practice, systems get around the problem with a loop that processes each document page by page, resetting the cache after every step.
Baidu derives its architecture from how humans copy text by hand. The DeepEncoder compresses pages while the MoE decoder processes them with R-SWA, running the KV cache as a fixed-length queue. | Image: Baidu
Baidu frames the problem with a human analogy. Someone copying a book doesn't re-read everything they've already written. They keep their eyes on the source, the last few characters they wrote, and the next one to put down. Older passages fade through a kind of soft forgetting. The researchers want Unlimited OCR to mimic that pattern.
A fixed window caps memory use
It works through what the team calls Reference Sliding Window Attention (R-SWA). Each generated token still sees all reference tokens, the visual image tokens and the prompt. But when it comes to previously generated output, it only looks back at the last 128 tokens. That keeps the KV cache constant throughout the entire process instead of growing linearly with output length.
