Issue with low-dimensional vector embeddings

Support
,
1

I have a model that produces low-dimensional vector embeddings - i.e. as low as 6 dimensions.

When I upsert this to the database, I can perform successful queries that retrieve similar results using Euclidean distance for around 10 minutes.

After this point, it no longer returns similar results. For at least the first 500 results, the “Score” attribute is 0, which I believe indicates it is an exact match. This is even though the vectors are noticeably different, and had previously returned a much greater “Score”.

Is this a known issue? It seems like a background optimisation is damaging the ability to retrieve data.

Any help in this area would be greatly appreciated.

2

Hi @matt1, and welcome to the Pinecone community forums!

Thank you for your question.

While I understand your reported issue clearly enough from your description, it would still help to see your relevant code (data pre-processing + upsert and query) if you would please share that.

In the meantime, I’ll ensure the correct team has received your inquiry.

Best,
Zack

3

Hi Zach,

Thanks for following up. After some experimentation, I have concluded that the model is likely not the best fit for Pinecone’s database and have switched to a custom database of my own.

Thanks for your help,
Matt