We are reviewing Pinecone as a solution for a vector database to store Embeddings of user messages.
My main question is how to store the vectors so we can query only specific user messages.
From what I read, there are two options.
Using meta data or using namespaces.
Can anyone share pros/cons for each of those options?
If you use namespaces, you won’t be able to query across different namespaces. So I would say go for them only if you want a stringent division between users here. Otherwise, metadata is a good option. Be careful, as very high cardinality on metadata will come with performance costs and consume more resources in an index.
I do want a complete separation between the users so seams that namespace should be the way to go.
Is there any limit on the number of namespaces I can create on a given index?
Would like to know the answer to “Is there any limit on the number of namespaces I can create on a given index?” as well. (i.e. is the index being optimized for horizontal or vertical scaling)?
There is no limit on the number of namespaces that you can create. A very large number of namespaces will also come with performance hits, but unlike metadata cardinality, it doesn’t creep up on you as you are always aware of the namespaces that you have