I’m storing vectors from multiple sources (A Wiki and a Ticketing system) in Pinecone.
I frequently re-index the sources independent from one another. So after processing the Wiki, I want to delete all old Wiki vectors and add all new ones. Upserting won’t be enough because some pages of the Wiki might have been deleted in the meantime and they should not be in the index anymore.
I want them to appear in the same query, and queries can only use one namespace, so splitting the vectors into one namespaces per source will not work.
So I’m left with metadata filters, e.g.: one metadata
sourcefield. But from the docs it looks like the delete function does not support filters.
That means I’d have to keep a state of all current ids (per source) locally and delete them by their ids, and hope that this never gets “out of sync”, which seems quite fragile.
Is there anything I’m missing or is deleting by metadata filter on the horizon?