I was on a call last month with a startup CTO who had just gotten their AWS bill. They had built a beautiful RAG application: semantic search, conversational AI, the works. Their vector index was humming along with about 50 million embeddings. Then they hit product-market fit.
Within six weeks, they scaled to 500 million vectors. Their monthly infrastructure costs went from $2,000 to $20,000. The real kicker? When we looked at the access patterns, over 80% of those vectors were queried less than once a week. They were paying hot-storage prices for data that was, by any honest measure, cold.
The standard advice here is "just use a cheaper vector database." The more interesting question is: why are you storing all your vectors at the same temperature in the first place?
The Cost-Recall-Latency Triangle
Vector search forces a three-way tradeoff. You can optimize for cost, recall, and latency, but you only get to pick two. Want high recall and low latency? That costs money (in-memory HNSW graphs with full-precision vectors eating RAM). Want high recall at low cost? Latency goes up. Want cheap and fast? Recall suffers.
