Vector similarity search makes massive datasets searchable in fractions of a second. Yet despite the brilliance and utility of this technology, often what seem to be the most straightforward problems are the most difficult to solve. Such as filtering.
Filtering takes the top place in being seemingly simple — but actually incredibly complex. Applying fast-but-accurate filters when performing a vector search (ie, nearest-neighbor search) on massive datasets is a surprisingly stubborn problem.
Hey,
Great article, thanks for writing it!
I’ve been thinking about the single stage filtering for a long time and I’m really interested in how you did it.
Do you intend to do another post explaining that?
Thanks
rom1504
Hey @rom1504 it’s on our “to-write” list which unfortunately sits below the “to-build” list. So no promises on when, but eventually yes! For now there’s no replacement for trying and seeing it for yourself.
This is a great article, but I have some little questions.
In Milvus, there is a hybrid search method that uses the metadata index to filter data first and get a bitmap about which vectors pass the filter, then just check if the vector is in the bitmap during normal ANN. I think this won’t be too slow.
By the way, might you give some idea about how you merge indexes into one? Is it through merging attributes into vectors?
No offense, just curious about this question.
Can you provide more technical details on single stage filtering .
Also curios is this single stage filtering applicable to a specific ANN algorithm like hnsw or faiss_ivf.
