Slava S. (slvDev) has optimized a 28.9M-parameter LLM running locally on an ESP32-S3 development board at around 9 tokens/s while generating text, or more exactly, telling short stories.We’ve written about LLMs and ESP32 in the past, but usually the boards are used as low-power gateways to interact with online LLMs hosted on powerful servers in data centers. Such products or projects include HiWonderLLM “smart chat module”, the ESP-Claw framework, or the ESP32 Agent Dev Kit. Slava’s “esp32-ai” project is quite different, as everything runs on the ESP32-S3 microcontroller.This won’t work on all ESP32-S3 as you’ll need enough memory and storage to store the 14.9MB (4-bit) model file, and Slava tested his demo on an ~$8 ESP32-S3 board with 512KB SRAM+8MB SPRAM and 16MB flash connected to a small I2C display to display short stories.Dave Bennett already did something similar with the ESP32-LLM project, but it only relied on a 260K-parameter model, far from the 28.9M-parameter model used in this project. The trick was to stop putting the model in “fast memory” (the 512KB SRAM), as most of a language model’s parameters sit in an embedding table with around 25 million rows, which can be read from slow flash. So only the few rows each token needs, or about 450 bytes, need to be loaded and processed in the SRAM.Slave explains that the idea is Google’s Per-Layer Embeddings used in Gemma 3n and Gemma 4. This is what the memory layout on the ESP32-S3 looks like: