Delete with metadata filters does not work

Support
1

Hi, I’m trying to use the delete method with namespace and metadata filters in the Python library and nothing happens. Here’s what I do exactly:

pinecone.init(api_key=key, environment=region)
index = pinecone.Index(<INDEX_NAME>)
delete_response = index.delete(namespace="96", filter={"file_pk": "16"})

# I have also tried this:
delete_response = index.delete(namespace="96", filter={"file_pk": {"$eq": "16"}})

When I make a query on namespace “96” I can see that file_pk exists:

{"content": "With anxious grap...",  "tokens": 202, "file_pk": "16"}

I can delete all records under a namespace with no problems but when I try with a filter nothing happens. The response from delete is {}.

Is this a bug or do I do something wrong? If it’s a bug, is there a workaround I could use until its fixed?

PS: I have also tried making a POST request to the API. Same results ({}).

Thanks in advance.

2

So does this make sense? Is this something that you guys are working on? Does it supposed to work like this?

3

Hi @Greg.C,

That’s the right way to delete by metadata. I just tried it myself and confirmed it works. In my case, I had a namespace called “example_namespace” and one of the vectors in it had a vector ID of “item_5”. I confirmed that vector had metadata setting “category: sports”, and delete with the filter worked.


>>> index.fetch(ids=["item_1"],namespace="example_namespace")
{'namespace': 'example_namespace',
 'vectors': {'item_1': {'id': 'item_1',
                        'metadata': {'category': 'music',
                                     'colors': ['purple', 'green', 'yellow'],
                                     'time_stamp': 0.1},
                        'values': <snip>
>>> index.delete(namespace='example_namespace',filter={"category":"music"})
{}
>>> index.fetch(ids=["item_1"],namespace="example_namespace")
{'namespace': 'example_namespace', 'vectors': {}}

Can you share the query you ran that returned that vector?

4

Same issue here, deleting vectors with metadata filters does not seem to work.

5

You can delete with metadata only on indexed metadata fields. Ensure that your index is properly created, e.g.:

metadata_config = {
    "indexed": ["file_pk", "another_indexed_field"]
}

pinecone.create_index("test", dimension=1536, metric="dotproduct",
                      metadata_config=metadata_config)
6

In this case, where the delete failed due to a field in the metadata filter not being indexed, can the API response indicate whether the vectors were deleted or not? Otherwise, there’s no way to distinguish a successful vs unsuccessful request.

7

How I create metadata_config for an existing index?

8

Hi @wiggawigga

I don’t think it is possible. The config has to be declared at index creation. If you already have an index I would create a collection, delete the index and create a new index from the collection with metadata_config.

Hope this helps

1 Like
9

Thanks. So I created a new index

 const pinecone = new PineconeClient();
  console.log("process.env.V2_PINECONE_API_KEY", process.env.V2_PINECONE_API_KEY)
    await pinecone.init({
      apiKey: process.env.V2_PINECONE_API_KEY,
      environment: process.env.PINECONE_ENVIRONMENT,
    });
    // Create index
    const createRequest = {
          name: "xxxxx",
          dimension: 1536,
          metric: "cosine",
          metadata_config: {
          indexed: ["id", "agent_id", "type", "url"],
  },
}
    await pinecone.createIndex({createRequest});

but it’s still doesn’t work. Here’s my code:

const client = new PineconeClient();
    await client.init({
      apiKey: process.env.V2_PINECONE_API_KEY,
      environment: process.env.PINECONE_ENVIRONMENT,
    });
    const pineconeIndex = client.Index(process.env.V2_PINECONE_INDEX);
    // delete record if existed
    const innerObject = {
  filter: {
    id: { $eq: webscrapeId }
  },
}

    await pineconeIndex._delete({innerObject});

    let url = steps.get_request.$return_value[0].url

    let metadata = { "id": webscrapeId,
      "agent_id": agent_id,
      "type": "URL",
      "url": url};
      
    console.log("metadata", metadata)

    const docs = `# ${pageTitle}
      ${pageDescription}
      ${markdownScraped}`

    const splitter = RecursiveCharacterTextSplitter.fromLanguage("markdown", {
      chunkSize: 500,
      chunkOverlap: 20,
    });
    const splitDocs = await splitter.createDocuments([docs]);

    let splitDocsFinalOutput = splitDocs.map(doc => {
      doc.metadata = { ...doc.metadata, ...metadata };
      return doc;
    });

    console.log(splitDocsFinalOutput);
    
    console.log("Size of splitDocs:", splitDocsFinalOutput.length);
    console.log("splitDocs", splitDocsFinalOutput[0]);

    // Insert into PineCone
    const insertPineCone = await PineconeStore.fromDocuments(
      splitDocsFinalOutput,
     new OpenAIEmbeddings(),
      {
        pineconeIndex,
        namespace: `${agent_id}`,
      }
    );

error that I’m getting:

PineconeError
PineconeClient: Error calling _delete: PineconeError: PineconeClient: Error calling _deleteRaw: RequiredError: Required parameter requestParameters.deleteRequest was null or undefined when calling _delete.
10

@wiggawigga

The current error has nothing to do with metadata which is good news. There is a problem with calling the method as one of the parameters is missing in your case :slight_smile:

Try this:

await pineconeIndex._delete({{innerObject}});

Hope this helps

1 Like
11

Apparently you need to add deleteRequest:

    const innerObject = {
  filter: {
    "id": { $eq: webscrapeId },
  },
}

    const deleteIndex = await pineconeIndex._delete({ deleteRequest: innerObject });

No error now but it still not deleting.

12

Can you fetch a vector with the same filter? Do you have the “id” field in your metadata?

filter: {
    "id": { $eq: webscrapeId },
  },
1 Like
13

Ok I finally got it working. here’s the code for anyone out there:

 const pineconeIndex = client.Index(process.env.V2_PINECONE_INDEX);
    // delete record if existed
    const innerObject = {
      namespace: agent_id,
  "filter": {
    "id": { $eq: webscrapeId },
  },
}

   const deleteIndex = await pineconeIndex._delete({ deleteRequest: innerObject });
  console.log("deleteIndex", deleteIndex)
1 Like
14

Happy to announce that Pinecone now supports delete-by-metadata on all indexes, including Serverless!