Incremental Upsert

Support
1

How can I upsert additional documents in future on the same namespace without overwriting or deleting the previous records or documents in that particular namespace?

The issue is whenever i want to upsert something new in the similar namespace, I’ve to upsert it with the previous files additionally to avoid overwriting

2

@ZacharyProser kindly look into this and let me know if that’s a possibility or a limitation atm.TIA.

3

Hi @taimoorqureshi80, and thanks for your question!

I understand your concern about upserting new documents without overwriting existing ones in the same namespace.

  1. Upsert Behavior: Pinecone’s upsert operation only overwrites records with the same ID. If you’re upserting new documents with unique IDs, they will be added to the namespace without affecting existing records.
  2. Generating Unique IDs: Ensure each new document you’re upserting has a unique ID. There are several ways you could achieve this:
    • Using a UUID or a similar unique identifier generator
    • Combining a timestamp with a document identifier
    • Incrementing a counter for each new document
  3. Partial Updates: If you need to update only part of an existing record, consider using the update operation instead of upsert. This allows you to modify specific fields without overwriting the entire record.
  4. Batching: You don’t need to include all previous documents in the upsert call when upserting new documents. Instead, you can upsert only the new documents in batches. We recommend batches of 100 or fewer records:

python

Copy

index.upsert(
  vectors=[
    {"id": "new_doc_1", "values": [...], "metadata": {...}},
    {"id": "new_doc_2", "values": [...], "metadata": {...}},
    # ... more new documents ...
  ],
  namespace="your_namespace"
)
  1. Checking Existing Records: If you’re unsure whether a record already exists, you can use the fetch operation to check before upserting. This allows you to decide whether to update an existing record or insert a new one.
  2. Namespace Management: If your use case allows, consider using different namespaces for different sets of documents. This can help organize your data and make it easier to manage updates.

I hope this helps!

Best,
Zack

4

Hey Zach,

Much appreciated. Can you let me know if we could search over different namespaces at the same time?

Any help would be appreciated.

Best Regards,
Taimoor Qureshi

5

Hi @taimoorqureshi80,

While it’s not possible to query multiple namespaces at the same time, you could implement a function that performs a query against a parameterized namespace, and then call this function concurrently using async syntax in your desired programming language.

For example, if you’re working in JavaScript, you could imagine having a function that queries one namespace, and then wrapping multiple invocations of that function in a Promise.all:

// This is sample code you may need to modify or adopt to your application
async function queryNamespace(namespace, query) {
    // Assuming you have already initialized the Pinecone client
    const index = pinecone.Index("your-index-name");
    
    const queryRequest = {
        vector: query.vector,
        topK: query.topK,
        includeValues: query.includeValues,
        includeMetadata: query.includeMetadata,
        namespace: namespace
    };

    return await index.query(queryRequest);
}

async function queryMultipleNamespaces(namespaces, query) {
    const queryPromises = namespaces.map(namespace => queryNamespace(namespace, query));
    return await Promise.all(queryPromises);
}

// Usage
const namespaces = ["namespace1", "namespace2", "namespace3"];
const query = {
    vector: [0.1, 0.2, 0.3],
    topK: 10,
    includeValues: true,
    includeMetadata: true
};

queryMultipleNamespaces(namespaces, query)
    .then(results => {
        results.forEach((result, index) => {
            console.log(`Results from namespace ${namespaces[index]}:`, result);
        });
    })
    .catch(error => console.error("Error querying namespaces:", error));

Or if you wanted to achieve something similar in Python:

# This is sample code you may need to modify or adopt to your application
import asyncio
from pinecone import Pinecone

async def query_namespace(pc, index_name, namespace, query):
    index = pc.Index(index_name)
    
    return index.query(
        vector=query['vector'],
        top_k=query['top_k'],
        include_values=query['include_values'],
        include_metadata=query['include_metadata'],
        namespace=namespace
    )

async def query_multiple_namespaces(pc, index_name, namespaces, query):
    tasks = [query_namespace(pc, index_name, namespace, query) for namespace in namespaces]
    return await asyncio.gather(*tasks)

# Usage
async def main():
    pc = Pinecone(api_key="your-api-key")
    index_name = "your-index-name"
    namespaces = ["namespace1", "namespace2", "namespace3"]
    query = {
        'vector': [0.1, 0.2, 0.3],
        'top_k': 10,
        'include_values': True,
        'include_metadata': True
    }

    results = await query_multiple_namespaces(pc, index_name, namespaces, query)
    
    for namespace, result in zip(namespaces, results):
        print(f"Results from namespace {namespace}:", result)

if __name__ == "__main__":
    asyncio.run(main())

I hope this helps!

Best,
Zack

6

Hey Zach,

We’re not able to retrieve appropriate chunks from our curriculum even after using metadata. Can you please assist us?

Best Regards,
Taimoor Hussain Qureshi