Upserting sparse values upsert a vector with empty sparse values

[louis]

Hi,

I am using the pinecone-text library, I am encoding my text using bm25, I got correct sparse values (same indices & values length).
I want to upsert those values in my pinecone pod (setup correctly for splade).
But when I do, the indices and values of all my vectors are empty.

I tried to upsert vector using Dict or Vector Object, but that does not change anything.

Do you have any idea why ?

I have a dimension of 1536.
The dimension of my indices and values are not 1536, this is not an issue right?

An example of the vector I’m upserting to pinecone:

{'id': '424ba0ac-c793-4b4e-8e38-cd8e18f57d23_3', 'values': [-0.0024523260071873665, [+1535 more]], 'sparse_values': {'indices': [4241179404, [+43 more]], 'values': [0.5157699443413729, [+43 more]]}, 'metadata': {[PRIVATE_METADATA]}}

[AudreyLorb.Pinecone]

Hi, @louis!

Audrey from Pinecone here .

Are you also using upserting a dense vector? Currently, Pinecone hybrid indexes must include a dense vector (hence their “hybrid” name). If you only want to query your sparse vectors, a common workaround is to upload a dense vector (of 1536 size, in your case) of all zeros.

Some Python code to generate a vector of 1536 dimensions that’s filled with 0s that would be something like:

import numpy as np
vector = np.zeros(1536)

If you need help “attaching” this vector to your sparse vector, so that you have a singular object to upsert into a Pinecone hybrid index, LMK!