Hi everyone, lately I’ve been having some trouble getting good results. I am developing an assistant based on pinecone and openAI that is responsible for answering questions to documents, the text is extracted from the documents, fragmented and through the use of OpenAI embeds it is transformed to later be inserted into an index in Pinecone. I have made sure that no pieces of text are lost during this process. There is a specific case when my query is about some alarm codes of a tool (the documents are manual) where I do not get the expected response even using words that are the same as the text fragment I want to get. I have noticed that the vectors that I want to obtain start to be part of the query result when I increase the top_k parameter but when using similar or the same words I don’t understand why the score is so low. I tried using other metric but I got the same result
Here some configuration data that I use:
Index:
metric: cosine
dimension: 1536
query:
top_k: 20