Local LLM inference has an expensive habit: It recomputes prefixes it has already seen. A system...
Local LLM inference has an expensive habit:
It recomputes prefixes it has already seen.
A system prompt.
A reused RAG document.
A few-shot block.
Block-hash and radix-tree prefix caching in vLLM and SGLang — when it actually saves prefill cost, and the eviction policies that…
TL;DR: We turned on vLLM's prefix cache for our agent workloads at Nexus Labs and watched TTFT drop...
Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted...
Chunked prefill in vLLM explains why a single 32K-token prompt stalls every in-flight generation — and how to tune the TTFT vs…
Learn how prefill and decode phases affect LLM app speed, what drives TTFT and inter-token latency, and which optimizations fix…
Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every...
