I need to display the results in pages of N documents and show the user how many total results are available
Is there a way to get use skip like other databases like mongo?
Is there a way to get histogram of scores? eg
scores 1-0.9 (15 results)
scores 0.9 - 0.8 (100 results)
…
This way I can show the user number of results with a certain similarity
Right now I am just asking Pinecone for top_k = 10000 and skiping and grouping this by myself … this is not ideal as it takes time and waste lots of data transfers since user doesn’t need all 10K every just wants to see the results distribution to them tweak the initial query_string
My app is helping users search through >10M docs using groups of keywords at a time