LLMs on Consumer Hardware — Part 2: Prefill and the Failure of the AI PC

Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model.

The two phases of inference

Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large.

The machines, and how they were measured

Machine

LLMs on Consumer Hardware — Part 2: Prefill and the Failure of the AI PC

Related reading

LLMs on Consumer Hardware — Part 1: The Stack and First Benchmarks

Running LLMs Locally on Consumer Hardware — Part 1: The Stack and First…

Running LLMs Locally in 2026: The Complete Guide to Benefits, Trade-offs, and…

Running Local LLM - 0$ Personal Agentic AI Assistant - Part 3

Comparing LLM Inference APIs: Cost, Performance, and More

Optimizing LLM Model Performance: Best Practices and Techniques