Just to kick things off here’s something a lot of customers have asked for: import/export of vectors. If this is something that would improve your use of Pinecone let us know by upvoting this comment. If you have any specific requests or ideas on how to implement it, be sure to leave your thoughts down below.
In my opinion, export is the larger need here as I think the existing batch upsert APIs are already sufficient to bulk import data.
On the export API I think there are a few obvious needs, which can likely just slot in to the Fetch endpoint.
- Retrieve all data, optionally limiting to a namespace and/or a matching metadata filter
- Flags for including metadata and/or values in response
Additionally, I think pagination is really important to consider, including consistent response ordering and limit/offset params for paging through the data. Also, how to support clients in retrieving all data from an index that is being actively written to.
2 Likes
Those are great ideas @silas! Would you mind creating a new post for pagination specifically? I think it deserves its own focus.
Done
And here are some extra characters to get beyond 20.
I have yet to try it, but there’s a free open-source tool that can do that now, it’s called Vector Admin. Seems quite powerful, can clone namespaces, indexes, can expoet/import to/from Pinecone/Chromadb for now…
Can also upload new docs, or edit vectors/metadata iirc.
So maybe that’s not a pressing feature now that there’s an alternative. Just saying.
Yep, I’ve played around with VectorAdmin and it’s definitely a nice tool with lots of potential for growth as a vector management tool. (Thanks to @tim for putting it together and open sourcing!)
But I think export is such a fundamental feature that it shouldn’t require paying for or running a 3rd party tool to accomplish it.
3 Likes
Hey, quick question,
whats the ETA for this feature (if any) ?