First of all, it is important to note that Pinecone only supports hybrid search with dot product as the similarity metric. This is the reason behind the error you encountered. To achieve normalized scores with hybrid search that incorporates BM25, here’s what you can do:
Dense Vectors: use an embedding model that yields normalized scores. This way, you won’t need the cosine metric. For instance, OpenAI’s ada-002 is already normalized, and many sentence transformer models can be configured to produce such scores.
BM25 Values: If you’re using the pinecone-text library, the BM25 scores are already normalized between 0-1. However, it’s worth noting that the nature of BM25 scores differs from that of dense vectors. While embedding models are designed to generate scores ranging from 0 to 1 - reflecting the likelihood of a document’s relevance to the query - BM25 normalization merely divides by the maximum possible score a document can achieve for a given query. This means the score distribution for BM25 might vary significantly from one query to another.
It’s worth noting that the convex combination of hybrid scores is basically a heuristic and may behave differently for individual use cases. The best approach is to experiment with your data and see what works well. If your objective behind normalization is to filter results based on a certain threshold, you might want to consider using a re-ranker, and filter results based on its logits.