I spent time debugging a RAG pipeline that looked correct in staging and behaved differently under real load. The job board I built processes thousands of listings every day, scoring each one against candidate profiles using LLM function calls. Getting that pipeline stable and affordable took more than a good vector store choice. It took understanding where the system actually breaks.
Here's what I learned about chunking, embeddings, vector stores, cost control, and observability when RAG stops being a demo and starts being a production service.
Chunking Strategy: The Decision That Determines Everything
Most tutorials treat chunking as a parameter you tune later. In production, your chunking strategy determines your embedding quality, your retrieval accuracy, and your cost structure from day one.
For job listings, I tried three approaches before landing on one that worked:
