Here's a dirty secret of search: "the closest match" and "the most useful result" are not the same thing. Return the mathematically nearest document and you'll often hand someone something technically related and practically useless. Relevance is a harder problem than similarity — and it's where good search is won or lost.
Getting that right was the core challenge in the GovernAI Research Atlas, a semantic discovery platform I built to unify research across papers, repositories, and policy.
Similarity is not relevance
Semantic search gives you a superpower: embed everything into vectors and find items close in meaning, not just wording. But raw nearest-neighbor retrieval has a blind spot. The vector-closest result might be a tangential paper that happens to share vocabulary, while the genuinely useful one sits slightly further out. Distance in embedding space is a proxy for relevance — a good one, but not the whole story.
If you stop at "closest vector," your search is clever and still frustrating.
