Training a 2.7B MoE from scratch for $200, one GPU at a time

Back to Articles

A few of us trained NanoColibri-Instruct — a 2.7B-parameter Mixture-of-Experts with 0.34B active params per token — from scratch, on rented GPUs, for about $180–260 total. No cluster: pretraining is sequential, so contributors took turns passing a baton, one GPU at a time. At its 5.4B-token budget it beats token-matched dense models on 5 of 7 zero-shot tasks with fewer active parameters than either baseline.

This post is the story, the numbers, the lessons (two of them embarrassing), and what's next. Everything is reproducible from the training repo: code, checkpoints, benchmark harness, and a public ledger of who trained what.

Why train a small MoE at all

The end goal isn't Nano — it's a class of models whose int4 containers are deliberately bigger than a consumer machine's memory, streaming experts from NVMe under a fixed RAM budget. A 16 GB laptop can't hold a 24–28B model in RAM; but in a fine-grained MoE, each token only touches the dense backbone, one always-resident shared expert, and a couple of 3–4 MB routed experts. Keep the resident stack pinned, cache experts with an LRU, and let the disk serve the misses — that's the bet.