Deleted vectors are still returned in query results

Support
1

I gave up on trying to filtering by namespace. I just deleted everything that was in a new namespace I created since filtering by namespace was not working.

I have run

res = index.delete(deleteAll='true', namespace='templates')
print(res)

Which outputs
{} now, since I have run it multiple times.

However, when I query over my index, I get outputs that include vectors that I explicitly deleted.

    embedding = get_embedding(query, embedding_model)
    response = index.query(vector=embedding, top_k=10, include_metadata=True, namespace="")[
        "matches"
    ]

response: [{'id': '8',

'metadata': {'is_template': True},

'score': 0.863455474,

'values': []}, {'id': '4',

'metadata': {'is_template': True},

'score': 0.857818961,

'values': []}, {'id': '68',

'metadata': {'is_template': True},

'score': 0.852902949,

'values': []}, {'id': '83', 'metadata': {'is_template': True}, 'score': 0.852275, 'values': []}, {'id': '22',

'metadata': {'is_template': True},

'score': 0.852252901,

'values': []}, {'id': '18',

'metadata': {'is_template': True},

'score': 0.851780355,

'values': []}, {'id': '61',

'metadata': {'is_template': True},

'score': 0.849612057,

'values': []}, {'id': '46',

'metadata': {'is_template': True},

'score': 0.848732948,

'values': []}, {'id': '70',

'metadata': {'is_template': True},

'score': 0.847001195,

'values': []}, {'id': '12',

'metadata': {'is_template': True},

'score': 0.844094157,

'values': []}]

[{'id': '8',

'metadata': {'is_template': True},

'score': 0.863455474,

'values': []},

{'id': '4',

'metadata': {'is_template': True},

'score': 0.857818961,

'values': []},

{'id': '68',

'metadata': {'is_template': True},

'score': 0.852902949,

'values': []},

{'id': '83', 'metadata': {'is_template': True}, 'score': 0.852275, 'values': []},

{'id': '22',

'metadata': {'is_template': True},

'score': 0.852252901,

'values': []},

{'id': '18',

'metadata': {'is_template': True},

'score': 0.851780355,

'values': []},

{'id': '61',

'metadata': {'is_template': True},

'score': 0.849612057,

'values': []},

{'id': '46',

'metadata': {'is_template': True},

'score': 0.848732948,

'values': []},

{'id': '70',

'metadata': {'is_template': True},

'score': 0.847001195,

'values': []},

{'id': '12',

'metadata': {'is_template': True},

'score': 0.844094157,

'values': []}]

[{'id': '8',

'metadata': {'is_template': True},

'score': 0.863455474,

'values': []},

{'id': '4',

'metadata': {'is_template': True},

'score': 0.857818961,

'values': []},

{'id': '68',

'metadata': {'is_template': True},

'score': 0.852902949,

'values': []},

{'id': '83', 'metadata': {'is_template': True}, 'score': 0.852275, 'values': []},

{'id': '22',

'metadata': {'is_template': True},

'score': 0.852252901,

'values': []},

{'id': '18',

'metadata': {'is_template': True},

'score': 0.851780355,

'values': []},

{'id': '61',

'metadata': {'is_template': True},

'score': 0.849612057,

'values': []},

{'id': '46',

'metadata': {'is_template': True},

'score': 0.848732948,

'values': []},

{'id': '70',

'metadata': {'is_template': True},

'score': 0.847001195,

'values': []},

{'id': '12',

'metadata': {'is_template': True},

'score': 0.844094157,

'values': []}]

These are the aforementioned templates which I had deleted. They should be gone and no longer accessible by any query.

In the same script, when I run

print(index.describe_index_stats())

I get

{'dimension': 1536,

'index_fullness': 0.3,

'namespaces': {'': {'vector_count': 25633}},

'total_vector_count': 25633}

Which is indicating to me that the namespace “templates” was in fact deleted. Then why does it still get returned in the result query?

2

You must have added those same vectors to the "" namespace. That’s the only explanation.

We recommend against using "" as a namespace name. It’s better not to use the namespace value in the first place; that will restrict any queries to the null namespace, which I think you want to accomplish here. I think the problem is that you’ve set this zero-character length string to a namespace, so you’re getting results inconsistent with your expectations.

I’d recommend deleting the vectors in the "" namespace so you know they are not present, then upserting them again into a new templates namespace. And be sure to name your namespaces in the future, even if you use default as a name rather than an empty string. I suspect there’s something else in your code somewhere that has caused this to happen (maybe an uninitialized variable when setting the namespace). But using explicit namespaces for everything will avoid that kind of bug from creeping in.

3

Before I added any vectors to the “templates” namespace, I had 25633 vectors in the “” namespace. Then I added around 5180 vectors to the “templates” namespace and deleted them. I still only have 25633 vectors in the “” namespace as shown on pinecone dashboard and then I describe the index and get ‘namespaces’: {‘’: {‘vector_count’: 25633}}. How is it possible these extra 5k vectors are still hanging around?

I am just avoiding having to delete everything and start again but it may be the only sensible choice