Can't search for vectors by metadata via after attempting to delete them but they still exist in actual database

Support
,
1

I am using a serverless pinecone database, so I attempted to make my own script to delete entries via metadata. See:

    while True:
        # First, list all records with metadata tag
        all_elements = index.query(
            namespace=namespace,
            vector=[0] * 1536,
            filter={
                f"{metadata_key}": {"$eq": f"{metadata_value}"},
            },
            top_k=10000,
            include_metadata=True
        )
        # When no elements remain, break
        if len(all_elements['matches']) == 0:
            break
        # Delete elements in batches of 100 (can only delete 100 elements at a time)
        all_ids = [element_dict["id"] for element_dict in all_elements['matches']]
        batch_size = 100  
        for i in range(0, len(all_ids), batch_size):
            batch = all_ids[i:i + batch_size]  
            index.delete(ids=batch, namespace=namespace)

Yet running this on metadata has produced a very strange and frustrating bug. When I attempt to query my namespace with the metadata filter I used to delete them via the API, no results show up. However, when I browse on the web client with the same metadata filtetr, all the vectors still appear. Additionally, whenever I query via other methods on the API (embeddings, other metadata), I still see these vectors in my pinecone database. I need to actually delete these entries via metadata, but now I am unable to query them with the proper metadata filter . What is causing this bug and how can I delete these vectors?

2

Hi @dchristl, I encourage you to try randomizing your query vector on each iteration to scan for similar results throughout the vector space.

When running a sample query and viewing results in the console, the provided query vector, while still simplistic, has more precision than a vector of all zeroes.

Keep in mind that this approach will not offer a guaranteed solution that deletes all vectors matching the metadata filter. That said, varying the query vector and providing more realistic values in the embedding will yield better results.