A few weeks ago our team of about ten engineers decided we wanted the "AI pair programmer in the terminal" experience without paying per-seat API costs for every developer, and without sending our codebase to a third party for every single query.
The answer we landed on: run an open model (Qwen) on a single shared GPU box we already owned, and have everyone's laptop talk to it over a private network. No cloud inference bill, no code leaving our own hardware, one box serving the whole team.
This post is the "why we built it, what broke, what we learned" writeup.
The step-by-step setup instructions are at the bottom if you want to replicate it.
Why we did it this way
