(While I asked this question with the Pinecone AI support, I just want to fact-check its answer is correct…)
I am learning to use Vector DB and comparing a few options for our use case. I am wondering, if Pinecone supports somethings similar to the ‘dynamic fields’ in Solr (Dynamic Fields | Apache Solr Reference Guide 6.6)
The scenario is like this that over time, I may want to add new metadata fields to my new records. So suppose on the initial indexing operation, I added 10k records each have 5 metadata fields. Next time, I have a new batch of 2k records, they have the same 5 fields, but also, I want to add 2 additional new fields. It does not matter if the initial 10k records did not have these new fields.
Is this possible? Thanks
Hello and thanks for your question!
Yes, it is possible to update the metadata fields of existing items in your index - even if your initial items did not have the metadata fields you wish to add in your subsequent updates.
There’s an example at this link of adding a new metadata field to items in an existing Pinecone index.
For example, let’s say you wanted to update this existing item with a new metadata field, which initially has only the type and genre fields:
("id-3", [4., 2.], {"type": "doc", "genre": "drama"})
You could accomplish adding a new field like so:
index.update(id="id-3", set_metadata={"type": "web", "new": "true"})
Following this update, your item would now look like this:
("id-3", [4., 2.], {"type": "web", "genre": "drama", "new": "true"})
This operation is referred to as a partial update.
Hope that helps!
2 Likes