The first time my scoring pipeline ran against a full day's batch, it took 47 minutes and cost $86. The second run took three hours because half the requests hit rate limits and the retry logic was too aggressive. By the third day I had a queue of unprocessed listings growing faster than the system could score them.
That's when I learned the difference between a RAG demo and a production RAG pipeline.
The system I built processes over 10,000 job listings daily, extracts structured fields via GPT-4 function calling, scores each listing against candidate profiles, and serves results through a REST API. It's been running in production for months. Here's what the architecture actually looks like and where most tutorials skip the hard parts.
Chunking Strategy: Why Document Size Matters More Than You Think
The naive approach is to dump the entire job description into a single chunk and call it done. That works until you need to extract 18 structured fields from a 2,000-word posting that mixes requirements, responsibilities, and boilerplate legalese.
