The idea of running a local LLM (Large Language Model) has always appealed to me, especially concerning data privacy and cost control. However, when I first delved into this, I realized through my own experiences how misleading market claims like "a few GB of RAM is enough" can be. In real-world scenarios, running a 70B parameter model with 8GB of VRAM is only possible with significant optimizations, which come with certain trade-offs.
In this post, I will share my experiences, the problems I encountered, and the solutions I found, from hardware selection to optimization techniques for local LLMs. My goal is to offer a concrete, practical, and "good enough" perspective to anyone interested in this field. As we begin, we must remember that VRAM is the most critical part of this equation.
VRAM: The Heart of Local LLMs and Capacity Limits
At the core of running an LLM locally is keeping the model's weights in the GPU's VRAM. As the model size grows, the amount of VRAM it needs naturally increases. For example, a 7 billion parameter (7B) model in 16-bit float (FP16) format requires about 14GB of VRAM, while a 70B parameter model can demand up to 140GB. These values are far beyond the hardware owned by an average user.
