I can’t find any good answers to this anywhere, so I ask here, hopefully someone has experience using Langchain’s PineconeHybridSearchRetriever to perform hybrid search on a Pinecone index. My question is; using the object’s invoke() method to query an index, why does this method produces a different result (i.e. return different documents) than Pinecone’s own index.query() method? Don’t they both use the same hybrid search algorithm?
I have made sure to pass both sparse and dense vectors to index.query(), so that is performs hybrid search. I also made sure to use the same embeddings for sparse and dense values for both methods. The weighting alpha is set to 0.5 for the The PineconeHybridSearchRetriever, which I guess is also the default for the index.query() method of Pinecone.
LangChain is an open-source framework, so you can see for yourself how the PineconeHybridSearchRetriever class is implemented. I suggest that you compare this against your personal code and figure out what the differences are.
Correct me if I’m wrong but pinecone’s index.query() method doesn’t support alpha parameters directly.
Nothing on the query docs about alpha and according to this slightly old guide on hybrid search, it looks like the alpha values are scaled prior to calling index.query().
The vector scaling implementation is identical to Langchain’s PineconeHybridSearchRetriever (see implementation here) so the issue could potentially be that you aren’t scaling your vectors in your native pinecone query implementation.