I hope you’re all doing well. I am currently facing a challenge with one of our Pinecone indexes and would greatly appreciate your expertise in resolving this issue.
We have a large dataset indexed in Pinecone; however, there has been a mismatch in the similarity metric used during indexing. The index was created with cosine similarity, whereas the model we are using relies on the dot product for similarity calculations. This disparity may led to inconsistencies in our similarity search results.
Due to the significant amount of data already indexed, reindexing the entire dataset is not a viable solution considering our operational constraints. I’m reaching out to inquire if any community members have encountered a similar situation or if there are any advanced solutions, workarounds, or best practices that could help us update the similarity metric for the existing index without causing data loss or significant disruptions to our service.
Any guidance or support provided would be immensely valuable. If you have specific procedures, tools, or best practices in mind, please share your insights. We are open to any suggestions and are willing to provide additional details about our use case if necessary.
Thank you in advance for your time and assistance. Looking forward to your responses.
If you have a vector dataset already in Pinecone, you can migrate it to a new index with the right settings using the Collections feature without having to re-embed or re-upload the data. A collection is like a snapshot/backup of the vectors in your index.
The high level overview of the procedure is:
Disable “ingest” into your index (you can continue to allow reads/searches, but you want to pause inserting or updating any new data during this process).
Create a Collection from your index
Restore the Collection to a new index from the Actions buttons on the Collections page. (You’ll be able to select the similarity metric you want on the new index.)
Switch your app to use the new index for queries
Configure any ingest apps to point to the new Index, and you can unpause ingest now.
1.) After you create the collection, you could delete the original index before your restore from the collection. I DO NOT advise this method, as it incurs a little downtime, and is at risk of extending to a long downtime if you encounter any issues during the restore.
2.) A better approach would be to just complete the procedure I outlined twice to get back to the original name:
Excellent! Once you’ve wrapped it all up and if my input was helpful/correct, would appreciate if you could flag the solution. Feel free to reach out if you need anything else.