An endpoint to retrieve all of the vector IDs stored in the index without supplying a query vector first.
should have optional namespace parameter
PLEASE add this, including a way to filter by metadata. Currently users are forced to do absurd workarounds like looping random query vectors in hopes to collect all of the matching vectors, or just deleting and reindexing.
Please guys, my workaround is now taking 10-15 minutes to find all the records in my DB.
@nsartor thanks for commenting. Would you be able to tell me a bit more about your use case? How large if an index are you exporting, how often are you doing this, and why? Thanks!
Sure. Currently in the 100k, but growing fast, some 5-10k a day.
I have IDs from another database that are stored in pinecone as they are created/edited.
I have a daily batch process that makes sure the two dbs are aligned, as sometimes the 1by1 process fails.
Currently I have the random vector workaround, batching 6 calls in parallel and iterating until all the vectors are found.
For some reasons the last hundred or so are always impossible to find and forcing it to find them all would require more than 100 calls to do so.
Luckily it’s not a time sensitive task, but I’d rather not do all these useless calls.
Ok got it so it is avoid skew between your “source” database and Pinecone. We are trying to simplify the keeping Pinecone What sort of database is upstream (mongo, mysql, etc.). How do you do the syncing? Is there a reason that skew emerges?
Errors or temporary exceptions on the functions gathering the info to be saved in pinecone as embedding. And the unreliability of the trigger.