When running ANN search with filtering, can I get the stats (such as the number of visited nodes, used RAM etc.) to determine how effective the algorithm is for my problem without having to try the actual-size data? If so, could you please give or point to an example of obtaining such stats, as well as to the relevant documentation?
Hi @mgoldenbe. Pinecone is a fully managed service, so those kinds of hardware stats aren’t exposed. Pinecone is designed to be easy to use and easy to get started with; just add your vectors and go without having to fine-tune the search algorithm itself.
OK. Let me give you my case. Suppose that my application has 1K users and each one stores 1M vectors (the details of how the documents correspond to the vectors are irrelevant). I need to make sure that when a user queries his vectors, he gets the top 200 matches quickly (say under half-a-second) and that this works within my budget.
I want to create a smaller set using the free offer and measure performance. However, to extrapolate from that to a much larger number of users and documents, I need to find out how effective the filtering is. That is, how many vectors not belonging to the user of interest the algorithm encounters and discards. So, I am not really talking about hardware stats, but rather about the stats collected by the search algorithm.
If I cannot predict how the algorithm will scale, how can I trust to use it in production and risk having to transfer to another platform with the disruption of operations that goes with it?
I think you are over-optimizing for something that just won’t matter with a managed service like Pinecone. You cannot and won’t have access to those metrics for something like Pinecone.
Speaking from experience Pinecone can scale to far more than 1M vectors (try 10M) and still return results very quickly - only constrained by pod size and indexing.
Unfortunately, I came to discover that Pinecone is fast because it insists on loading the whole index into memory (albeit compressed), which makes it too expensive for my application. A user of my app might store 1M 384-dimensional vectors and pay $3/month. But the s1 pod can house only 10M such vectors and costs $80/month…
1 Like
Pinecone has taken the approach of abstracting away those details from the user in favor of operational simplicity, and yes that has cost implications. But like with any technology decision there are always tradeoffs, including cost, performance, complexity, operational overhead, etc. The good news is there are plenty of open source alternatives to Pinecone, so hopefully one of those will meet your needs.
Cheers,
Silas
2 Likes
Is there any other system that offers effective filtering?
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.