How We Built Instant Translation Help (即时翻译帮助) with Python and LLMs

Balancing speed and context with a hybrid glossary + LLM caching system

The Need for Instant Translation Help

At LectuLibre, we use LLMs to translate entire books into different languages. But even after a high-quality translation, readers sometimes stumble upon an unfamiliar word or want to see the original phrase for clarity. We envisioned a feature we called 即时翻译帮助 (instant translation help): clicking on any word in the translated text would immediately show a contextual explanation or alternative translation, right inside the reading interface.

The core requirement was speed. The popup had to feel instantaneous—under 500ms. A spinning wheel disrupts the reading flow. However, making a full LLM call for every click was out of the question: latency ranged from 2–5 seconds, and at scale it would be expensive.

Breaking Down the Problem