I’m having difficulty understanding the best approach to using Pinecone indices and namespaces. Based on the picture below, what is the best way to dynamically scale my use of namespaces as my user base grows.
Each index can have different embedding models and that is very useful, but in my implementation I am using the same embed model for all indices. What is the easiest way to create new namespaces for each user without having to verify if my application has used all available namespaces within an index?
For instance, in the Serverless tier:
If each user has an namespace, and I’ve reached 100 users within my 1st index, then the next user’s namespace should be in the 2nd index. What is the recommended implementation to change indices in this situation?
What I’ve tried is to create this logic:
- Check if index exists
- If it doesn’t exist, create a new index and create a new namespace by ingesting a file with fromDocuments(). Break
- If it exists, check the amount of namespace using DescribeIndexStats()
- If the number of namespaces is 100 and if there are 5 indices already, then show an error.
- If the number of namespaces is NOT 100 and if there are not 5 indices, then use the index from step 1 and create a new namespace by ingesting a file with fromDocuments(). Break
- If the number of namespaces is 100 and if there are not 5 indices, recurse to 1 with a new index id.
If I were to include “Projects” as well for Standard or Enterprise tiers this gets a bit more complicated but follows a similar logic.
Any other ideas would be most welcome.
Thanks!