The push to run large language models directly on phones and tablets is driven by three hard requirements: latency, privacy, and offline availability. But the physics of mobile hardware creates a ceiling. NPUs and DSPs on flagship SoCs are powerful, yet thermal design power and battery capacity turn long-context inference or multi-turn reasoning into a rapid drain. The practical path forward is not all-edge or all-cloud. It is a tiered architecture where small, quantized models handle sensitive, frequent tasks locally, and a predictable cloud API handles everything else.
Model Selection and Quantization for Mobile
To keep power draw under control, the model must fit into device DRAM without constant swapping, and the working set must be small enough to avoid sustained high-frequency memory clocks. For most current mobile hardware, this means targeting models between 1B and 4B parameters, quantized to INT4 or INT8.
Strong candidates include Llama 3.2 1B and 3B, Qwen 2.5 0.5B through 3B, Phi-3 Mini 3.8B, and Gemma 2B and 4B. These architectures use grouped-query attention or multi-query attention, which shrinks the KV cache and reduces memory bandwidth, one of the largest contributors to energy consumption on mobile SoCs.
