AirLLM: Running 70B Parameter LLMs on a Single 4GB GPU

Low-Memory LLM Inference: Meet AirLLM

As open-source Large Language Models (LLMs) continue to grow in capability, their hardware requirements have ballooned alongside them. Running a 70B or 405B parameter model traditionally demands enterprise cloud GPU servers equipped with hundreds of gigabytes of VRAM.

AirLLM is an open-source Python library developed by lyogavin to make massive model inference accessible on standard consumer hardware—allowing developers to run 70B models on GPUs with as little as 4GB of VRAM.

What is AirLLM?

Rather than attempting to fit an entire neural network into GPU memory at once, AirLLM uses a "divide and conquer" execution architecture. It streams individual model layers sequentially from disk into memory, computes the output for that specific layer, and then clears it before loading the next.