Does Pinecone support merging namespace?

Hi, I would like to know if its possible to merge multiple namespace together. And if not is there any other work around. I haven’t seen any documentation about similar topics, so I assume the only way might be copy all the vector from one namespace and insert it into the other one? Thanks in advance !

Hi @suzushi-tw and thank you for your question!

Pinecone does not yet natively support merging namespaces, but here’s some starter code for how you could go about doing that with the Pinecone Python client:

# Function to fetch all vectors from a namespace
def fetch_all_vectors(namespace):
    random_vecs = [random.random() for _ in range(1536)]
    response = index.query(
        namespace=namespace,
        vector=random_vecs,
        top_k=100, # Adjust for size of your dataset
        include_values=True
    )
    return response['matches']

# Fetch vectors from ns1 and ns2
vectors_ns1 = fetch_all_vectors("ns1")
vectors_ns2 = fetch_all_vectors("ns2")

# Convert fetched vectors to the required upsert format
def format_vectors_for_upsert(fetched_vectors):
    return [{"id": match['id'], "values": match['values']} for match in fetched_vectors]

formatted_vectors_ns1 = format_vectors_for_upsert(vectors_ns1)
formatted_vectors_ns2 = format_vectors_for_upsert(vectors_ns2)

print(f"Preparing to upsert {len(formatted_vectors_ns1)} from ns1 and {len(formatted_vectors_ns2)}")

# Upsert vectors into the merged namespace
upsert_vectors(formatted_vectors_ns1, 'merged')
upsert_vectors(formatted_vectors_ns2, 'merged')

print(f"Upserted {len(formatted_vectors_ns1)} vectors from ns1 and {len(formatted_vectors_ns2)} vectors from ns2 into {target_namespace}")

This code queries vectors out of namespaces: ns1 and ns2 and then upserts them to a new namespaced named ‘merged’.

You could use this as inspiration to do something similar against your own namespaces.

Hope that helps!

Best,
Zack

1 Like

Thanks a lot, got it working in typescript. Leaving this code snippet and hope it could help people looking for similar solutions :

           for (const id of selectedIds) {
                const pineconeIndex = pinecone.Index(process.env.PINECONE_Index!).namespace(id);
                const queryResponse = await pineconeIndex.query({
                    vector: new Array(1536).fill(0),
                    topK: 10000,
                    includeValues: true,
                    includeMetadata: true
                });
                console.log(queryResponse)

                const matches = queryResponse.matches;

                const filteredMatches = matches.map(match => ({
                    id: match.id,
                    values: match.values,
                    metadata: match.metadata
                }));

                console.log(filteredMatches);
                await pinecone.Index(process.env.PINECONE_Index!)
                    .namespace(newchat.id).upsert(filteredMatches)
            }
1 Like

@suzushi-tw,

Delightful! So glad to hear you got it working and THANK YOU for sharing the code snippet.

I’m considering putting together a Jupyter Notebook and may use your code for a TypeScript example.

Thanks for using Pinecone!

Best,
Zack

1 Like

Here’s our new Jupyter notebook for merging namespaces. Please note that it’s experimental and not guaranteed by Pinecone, so test thoroughly before using in production!

1 Like