Every system that does "semantic" anything — RAG pipelines, recommendation engines, image search, dedup — boils down to one operation: given this vector, find the closest ones out of millions. The vectors are embeddings, a few hundred to a couple thousand numbers each, and "closest" means closest in meaning.
You'd assume the database either scans all of them (slow but correct) or uses some clever tree to jump straight to the answer. It does neither. Instead it deliberately settles for the approximately closest vectors — and that compromise is the entire reason vector search is fast enough to exist. Two algorithms do almost all the heavy lifting in practice, in pgvector, Qdrant, FAISS, and the rest: IVF and HNSW. Here's what they're actually doing under the hood, and how to choose between them.
Why "exact" is off the table
The natural objection is: why approximate? Just find the real nearest neighbor. In two or three dimensions you could — a k-d tree or similar structure prunes away big regions of space and finds the true closest point quickly. The trouble is that embeddings live in hundreds of dimensions, and high-dimensional space is deeply weird.
It's called the curse of dimensionality. As dimensions grow, the distance to your nearest point and the distance to your farthest point drift toward being almost the same. Formally, the contrast (d_max − d_min) / d_min shrinks toward zero. When everything is roughly equidistant from everything else, a tree can't confidently say "skip this whole branch, it's too far" — the bounding regions all overlap, every branch looks plausible, and the search degrades into checking nearly everything. Exact indexes quietly collapse back into brute force.
