600 MiB of headroom
My RTX 4070 was running Qwen 35B beautifully after the --cpu-moe trick from a previous run. The tokens/sec were where I wanted them. VRAM sat at 11,714 MiB out of 12,281 — 95% full.
That leaves 600 MiB. Not enough for a serious agent.
The context window I was giving llama.cpp was -c 4096. Fine for chat. Not fine when a Claude Code-style agent hands the model 12,000 tokens of tool definitions before it says hello.
I wanted -c 32768. That's an 8× jump. And the memory that grows with context length is the KV cache. Multiply the cache by 8 with 600 MiB free, and llama.cpp dies during warm-up. I know because I tried it first.
