A practical field guide to running open-weight LLMs on your own hardware — which models, which serving stacks, and the cost math nobody puts on a blog post.
A fintech client called me in January with a problem that is getting more common by the month: their compliance team had banned every external AI API. No data leaves the building, they said. The analyst who wanted to chat with their internal documents would just have to wait. I told them they did not have to wait — they could run the model on hardware they already owned.
Three weeks later we had a 70B-class model serving the entire company on two GPU boxes, handling their document Q&A, their support triage, and a chunk of their internal code review. The cost per million tokens was a fraction of what the hosted API would have billed, the data never left the office, and the compliance team slept better. This article is that deployment, generalized: how I think about open-weight models, which ones I actually reach for, how I serve them, what the hardware math looks like, and the honest list of things that go wrong.
Why Self-Host at All
Let me state the case plainly, because there are good reasons and bad reasons to self-host, and you should know which one you have.
