Fetching data by a list of IDs efficiency issue

General
, ,
1

Hi Pinecone community,

I want to retrieve data using a list of IDs (not necessarily all IDs, just a random subset) in an index. I tried Fetch data - Pinecone Docs. However, I noticed that it is relatively slow compared with querying (I guess because the vector values are also included? And there is no way to exclude that). Is there an efficient way to quickly retrieve data using a list of IDs?

Thanks!

2

Hi @Max123, I have a few questions aimed to help clarify what you are trying to accomplish:

  1. About how many records do you intend to retrieve?
  2. Do you always know the exact IDs you want to retrieve, or can the “random subset” actually be random?
  3. What data are you trying to retrieve if you do not care for the vector data?

Looking forward to hearing back from you!

3

Hi @zeke_pinecone ,

The situation is that we have two services that require the same search results. Initially, I was using two identical vector searches to obtain the data. However, since the search results are not guaranteed to be identical (as discussed here Pinecone return results consistency - Search, No Filter - Pinecone Community), I plan to conduct a search in the first service and retrieve the data using IDs in the second service.

  1. The records I aim to retrieve are from 500 to 2000. The database size is 100k to 500k (I give a range because it is not finalized)
  2. I have the exact IDs.
  3. Please refer to the above situation.

Thank you very much for your help!
Max