Hybrid (dense + sparse) embeddings and index dimensions

Search, No Filter
1

Hi,
I am reading some docs and also see the deepai video, I am bit confused as to what should be the dimension value of a index that is used for dense and sparse vectors

  1. Can it be created with no dimensions?
  2. Has to be of dense vector dimensions?
  3. Can the dense and sparse dimensions be different?
    Thanks in advance
  • Anu
2

I think the answer is as below from Upsert sparse-dense vectors?
Pinecone supports sparse vectors of up to 1000 non-zero values and 4.2 billion dimensions.

Assuming a dense vector component with 768 dimensions, Pinecone supports roughly 2.8M sparse vectors per s1 pod or 900k per p1 pod.

can some one please confirm if any changes?

  • Anu
3

Index dimension should be equal to dimension of vector for dense embeddings