On-premise RAG without GPU, cloud, or Docker: five lessons that cost me a week each

Every RAG tutorial I've read makes the same two assumptions: you have a GPU, and you can call a cloud API. For the environments I build for, both assumptions are wrong.

I work on health information systems in the public sector. The stack has to run inside institutional infrastructure — no data leaves the network — and the hardware I get is whatever the procurement cycle produced two years ago. In practice that means Windows Server, CPU only, and open-weight models running locally.

So I built a RAG stack that runs entirely on-premise, no GPU, no cloud, no Docker. It's open source at github.com/psychohub/rag-onpremise: ASP.NET Core 9 for orchestration, Ollama for local inference, Qdrant for vectors, Python for the ingest pipeline, Mistral 7B as the LLM, nomic-embed-text for embeddings.

Getting it into production took longer than the design did, because five things broke that no tutorial had warned me about. This is the field report.

The environment, and why it matters

On-premise RAG without GPU, cloud, or Docker: five lessons that cost me a week each

Related reading

I Built a Production RAG System on My M1 Mac for $0

Building a Fully-Local Research RAG on 2 GTX 1080 Ti + an RTX 3090 — 3 Gotchas

RAG isn't an AI problem. It's a data engineering problem wearing an AI hat.

Anatomy of a Full RAG Application: Every Concept, One Self-Hosted Stack

What's the first thing that breaks when a RAG system leaves the notebook?

Building Production-Ready RAG Applications: A Practical Guide