I am new to Pinecone and Semantic Search. We currently use Elasticsearch for our search needs, but we are evaluating semantic search.
I indexed 5 documents (Pinecone index says: Total Vectors - 5), but when I search the index with the query “xxxxxxxx” or any random text, it returns all 5 ids in the response. What am I doing wrong?
I am creating vector embeddings for both indexed data and input query via OpenAI text-embedding-ada-002
model -
We are using the following ruby gem for accessing Pinecode API - GitHub - ScotterC/pinecone: Ruby client for Pinecone Vector DB
Here’s the query with options…
pinecone_index.query(
{
vector: open_ai_vector_embeddings(query),
namespace: PINCODE_DEFAULT_NAMESPACE,
top_k: 10,
include_values: false,
include_metadata: false
}
)
Here’s the document insert, where document_data
is a blob of text.
pinecone_index.upsert(
namespace: PINCODE_DEFAULT_NAMESPACE,
vectors: [
{
id: id,
values: open_ai_vector_embeddings(document_data)
}
]
)
Index dimension - 1536
Index metric - cosine