Bonsai-27B on a Single 3090: What Works and What Doesn't

I spent the weekend with Bonsai-27B, and I think I finally understand what a "27B that runs on one GPU" actually means in practice.

The GGUF quants from prism-ml landed on HF this week, so I pulled the Q4_K_M — about 16GB, fits comfortably on a single RTX 3090 with room for a 16K context window. No sharding, no tensor parallelism, no tricks. Just llama.cpp and a model file. That alone is worth noting because most models that claim "single GPU" actually mean "single GPU with 48GB" or "single GPU if you quantize to 2-bit and squint." This one genuinely works on consumer hardware.

First thing I noticed: it's fast. On my machine I'm getting ~28 tok/s at 4K context, which is usable for chat and fast enough for batch classification work. At 16K it drops to about 19 tok/s — still fine for interactive use. The MoE architecture (35B total parameters, 3B active per token) means you're effectively running a 3B model's compute budget while keeping a 27B's worth of learned parameters in memory. That trade-off is the whole reason this model exists, and it mostly delivers.

I threw a few real tasks at it. Structured JSON extraction from messy invoices — solid, no hallucinations on field names, handled edge cases like missing fields gracefully. A short RAG pipeline over some internal docs — it retrieved and synthesized cleanly at 8K context, though I noticed it occasionally skipped a relevant chunk when the context got dense. That's a known MoE behavior: the router can get confused when too many similar documents compete for attention in the same window.