Generally, when a user asks a query, the system searches for the relevant chunks stored in the vector database using cosine similarity. The better we can filter the data, the smaller the search space becomes, resulting in faster and more efficient retrieval.
Suppose we have a book with 10 chapters. If we want to search for a particular topic, all the points in the vector database are compared with the user query, and only the closest points are retrieved. This process is called KNN (K-Nearest Neighbors).
Another algorithm is ANN (Approximate Nearest Neighbors). Instead of checking all the points in the vector database, ANN searches only within a smaller region based on the proximity of the data. As the name suggests, it does not always return the exact result, but it provides the most preferred or approximate results much faster.
Is there any other method we can use to make the search more effective?
Metadata Filtering
