I am developing a RAG application using Pinecone Serverless. In this setup, each user’s data is isolated within its own Namespace. Every user has multiple documents, each distinguished by unique metadata. Currently, my approach involves using metadata filters to search for specific documents. However, as the number of vectors per user increases, potentially reaching millions, the cost associated with “Read Units” is likely to rise substantially.
My concern is whether it’s possible to store documents in separate partitions within the same Namespace. Such a structure would mean that during a query, Pinecone wouldn’t need to scan the entire Namespace of a user, potentially reducing costs and improving efficiency.
An alternative I considered was creating a separate index for each user. Within each user’s personal index, their documents would be divided by Namespaces. However, this approach restricts the ability to perform queries across multiple user documents, as queries are limited to a single namespace.