Deleting vectors that match a metadata field is possible using the delete vector API by passing the filter parameter.
However, since this particular metadata field is high-cardinality (millions of values) and we have no need to query based on this value, we use selective metadata indexing to avoid indexing it:
My question is if the metadata field is not indexed, then is the delete-by-filter using that metadata field operation accurate and performant for large data sets.
Possibly related is this unresolved question; in reading this my concern is that the filter does not work if the metadata field is not indexed.
Hi @richie , the quick answer is “no, you can’t delete by metadata on a field that is not indexed.”
The longer answer is that delete by metadata depends on the metadata filters to find the relevant vectors to delete. If those fields aren’t indexed, the engine can’t find them, so the delete won’t work. It’ll essentially end up with an empty set of vectors to delete, so won’t delete anything.
What is the field in question? And is it a single field or more than one?
Could you use that field as the vector ID instead? This would both solve the problem of cardinality, and allow you to fetch/delete by ID just the vectors you need that match that field’s value.
I think, for our purposes, that we need to store a map from the metadata field to pinecone vector IDs in a separate key-multivalue store, and manage that.