TL;DR— Running open-weight models locally shifts the real cost from API bills to evaluation debt. Quantization, engine choice, and version churn all silently change model behavior, and most teams never re-test after making these changes. The hard part of local AI isn't installing weights— it's building the continuous eval pipeline that vendors used to run for you.
Every pitch for open-weight models ends the same way: download the weights, point an inference engine at them, and you're free. No rate limits, no vendor lock-in, no surprise deprecations. What that pitch leaves out is that the vendor was doing something for you besides serving tokens. They were running eval suites against every change before it shipped. The moment you self-host, that job doesn't disappear. It becomes yours, and almost nobody budgets for it.
Quantization Is Compression, and Compression Is Lossy
The whole reason local AI is viable on normal hardware is quantization. A dense model at full precision needs GPU memory most teams don't have; a 4-bit or 8-bit quantized version of the same weights fits on a single card. Formats like GGUF, AWQ, and GPTQ made this almost mechanical— pick a scheme, run a script, get a smaller model that mostly behaves the same.
