Oh great people of the internet. I have a problem I could use input on.
Let’s say I have several documents stored in an index for a user. These documents could be A, B, C and D. That user then has two different LangChain agents. They want those agents to have different knowledge stores related to those 4 documents. Imagine Agent 1 needs access to A, B and C documents and Agent 2 needs access to only C and D documents.
What’s the best way to do this? This is a simplified example as we will have many users, many documents and many agents so I’m looking for a solution that makes the most sense at scale.
Would it be best to:
Use namespaces somehow to segregate?
Use Meta Data that I can query on?
Duplicate the Documents altogether and have each agent access its own copy?
My proposed solution involves fine-tuning a model specifically to generate accurate namespaces and metadata based on the input question.
The feedback loop could be the refine QA method langchain uses or the sequential method.
The only possible downside with this is the amount of token that you need to spend. The finetune models is of course a bit less expensive than gpt-4 so i would suggest you look down that road!
Fingers crosses for lower api prices at November 6 at dev day
Feel free to write and ask any question if you have some
Fair i though you had a basic understanding of how a vector database is working
The finetuned model can be used to extract the namespace and metadata filter from the query. This way the agents would have a higher change of choosing the correct namespace, without the user have to state witch part of the index they want to retrieve the information from.
So i see no reason not to use this option if you end up having multiple namespaces.
It seems like you have different documents that contains information from different mediums, but without knowing the context of the document it is 100% impossible to give an answer… stating A, B and C gives us no basis to answer the question…
I recommend familiarizing yourself with the fundamentals of vector databases and the techniques for retrieving vectors from extensive datasets.
Using metadata/filtering is fine for creating overlapping sub-sets of data that apply to different agents, assuming that each agent already knows which dataset is relevant.
You can specify the metadata filter at query time and Pinecone will limit the scope of the search based on that filter.
Where you’ll most likely end up with some complexity is around the indexing of content and keeping the metadata up to date, particularly if ad-hoc user actions drive changes that affect which data comes into or out of scope for an agent.
For example, I’m guessing you’ll find yourself wanting to do things like update a metadata flag for all vectors that match some other metadata filter, and that kind of thing is not supported in the API yet. Pretty much all of the management APIs are based on already having the IDs of the rows you want to make a change to. There are some workarounds to get those IDs, but things just might take a few more steps than you might expect.
Good luck, and let us know if we can be of any further help.
Would it be best to apply metadata this way? Metadata would need to be changed as well. User 1 may decide, for example, to apply Bot 2 to vectors belonging to Document B at a later date. In this case we would need to update bots metadata for those vectors to be an array with [1,2] as the value.
Can you query metadata by values within a key/value array as shown?
You can query by metadata values with arrays and you can find the details here: Filtering with metadata
Just checking, in your example do the docs A, B, C, and D apply to all users?
Or are ABCD scoped to a single user, let’s call them User 1?
If User 2 comes along, would they have their own separate docs (EFGH) and different bot IDs that map to those EFGH docs?
Got it, then I’d also suggest using namespaces to isolate user data (namespace per user). There are no limits to the number of namespaces you can create. That’ll make it easier to access and manage the data for a single user, and is conceptually pretty easy to reason about as you develop your app.
Oh you’re creating a separate index per user? That could get very expensive depending on the number of users and depending on how much your users are paying you for your service. Minimum $70 per index per month.
The main reason to deploy separate indexes are if you need to store vectors with different dimensions or use a different similarity metric (cosine vs. euclidean, etc.).
Got it! So we would use one index. What would be a reason of needing documents to have differing dimensions? Documents with differing data structures like CSV? Also, if you’re open to a contract role I’d be happy to talk about paying for your time.
