Getting a small local language model running on a notebook or even smartphone in 2026 is trivial. But what about something even smaller and lower-power. Say, like an ESP32 microcontroller that costs less than $10?It might sound impossible — the device is primarily designed for things like remote sensors, IoT, and other embedded applications, not running generative AI models — yet, that's exactly what a developer who goes by the handle SlvDev has managed to do.In a process detailed on GitHub, and recently showcased on the Better Stack YouTube channel, SlvDev documented how he managed to get a small language model running at nearly 10 tokens a second locally on a microcontroller that costs about the same as a fancy cup of coffee.
Tiny stories on a tiny microcontroller
Cramming a large language model (LLM) onto something as small as a ESP32 microcontroller isn't a trivial task. There's a reason that these models are trained and run on GPUs. LLMs are memory-hungry beasts that typically require between one and four bytes per parameter just to hold their weights in memory.With just 520 KB of SRAM and 8 MB of pseudo SRAM (PSRAM) on the ESP32-S3, you aren't going to be running a model like DeepSeek V4 Flash .To make it work, the dev had to drop the "large" from the language model and settle for something nearly 10,000 times smaller: TinyStories, a 28.9 million-parameter model originally developed by Microsoft Research.However, even this model is asking a lot of an ESP32-S3 module. At 16-bit precision, the model requires about 60 MB of memory that the ESP32 simply doesn't have. So, the dev employed several techniques, some of which we've previously explored, to shrink the model’s footprint.The first is quantization, a process by which weights are compressed by reducing their precision from something like 16-bits of precision to eight, or even four.This enabled SlvDev to trade a bit of accuracy for a 75 percent reduction in memory required. Instead of about 60 MB of memory to hold the weights, they now require just 14.9 MB. But that still wasn’t enough. Thankfully, in addition to the 8.5 MB of working memory, the ESP32-S3 can also be bought with up to 16MB of flash storage.
