Serverless index intermittently fails to return anything to queries

We moved our index to serverless version about a month ago. It has been working fine until this week. For about 4 days now it intermittently fails to return any matches to queries. And then a couple hours later without us changing anything on our end it will again start returning matches.
I have also tried performing the same query also from UI and during these periods of downtime there are also no matches (or just one always the same match) in UI.
Our query is standard dense vector search with two additional metadate filter values (env and user_id).

Any suggestions what could be the issue here?

1 Like

@Janis Can you please share the name of the index you’re querying? We recently had an incident impacting queries, which is documented on our status page. I’d like to check your index in question with our engineering team and ensure there is no lasting impact.

We are also having problems with the query function returning either an empty set of matches or a limited number of matches for the same exact filter at random. However, it occasionally functions as expected.

1 Like

Hi Zeke, thank you for the prompt answer.

The index:
vennie-serverless-1
Host slug:
vennie-serverless-1-74d7259

The query:

  • dense vector query
  • Metadata filter:
  • environment == dev / prod
  • user_id == <user_id>

Hello @zeke_pinecone, thank you for the help with this issue. My team is also having the same issue. Could you also look at our index to see if it’s a persistent problem?

I’m also having this exact issue, it seems that nothing is returned if the index hasn’t been queried in a while. Then once it starts working after a few queries have been made, it starts returning results for any query. How can I fix this issue? I don’t want users of my app to receive nothing when they first try to search my index.

@janis Thanks for these details. In your initial message, you stated:

For about 4 days now it intermittently fails to return any matches to queries. And then a couple hours later without us changing anything on our end it will again start returning matches.

Can you please confirm that in the instances where the query did return results, you were using the exact same metadata filter as when it did not? In other words, I’m seeking to validate that you have records in your index that match the filter and are recognized by our query routers when the expected behavior occurs.

Additionally, providing a timestamp (including the timezone) of a query that returns no results would be helpful.

@katie @SailingSF @Janis

If you have not done so already, please open a support ticket using the email address associated with your Pinecone account.

Please be sure to reference this forum post and include the name of the index in question, sample queries, and any timestamps of queries returning zero results. Thank you!

I can confirm we were using the exact same filters and tags. The same issue occurred today and after approximately 10 minutes the index was back up and working as usual. I will try and find the timestamp.

Hello all,

We have rolled out a fix to address this issue.

Please let us know if you’re still seeing this problem.

Thanks

1 Like

My ticket is still open but to put it on this forum for anyone I have seen no change in the issue. Queries still return no results for the first 10 or so attempts of the same exact query.

For me as well, first few queries return nothing or incomplete results. Then, after a while it starts returning correct results for the exact same queries. This has been happening since Thursday.

Hello @SailingSF and @cesar,

A further fix was rolled out yesterday.

Are you still seeing this problem today? If you are please let me know the timestamp it last happened and the index host.

Thanks,
Patrick

Hi Patrick,

The last times this happened were just a couple hours ago:

Mon, 29 Jul 2024 04:32:38 GMT
https://ai-in-je-website-5b00p64.svc.aped-4627-b74a.pinecone.io
us-east-1

I clonedmy index to another region and it happened there as well:

https://ai-in-je-website-ireland-5b00p64.svc.apu-57e2-42f6.pinecone.io
Mon, 29 Jul 2024 09:40:22 GMT
eu-west-1

Appreciate you looking into this. I also submitted a ticket via the support form.

Thanks!

Cesar

@cesar Thank you for the details. I have also reply to your ticket.

@cesar Can you confirm if you’re seeing it continually or only intermittently?

@cesar One more follow-up question: the timezone is GMT. Did you mean UTC or BST? As GMT is BST right now.

Hello all,

I wanted to give an update on this situation. We rolled out a further fix on Monday. Since then we have not had further reports of this problem

Please do let us know. if you are still seeing this issue.

Thank you for your patience and for working with us on this.

Thanks,
Patrick

Hello. I am having this issue as well. In the UI, if I go to a namespace that I haven’t queried before, the first query returns zero results. The next attempt is successful. Why is this happening?

Metric: cosine

Dimensions: 1024

Cloud: GCP

Region: us-central1

Type: Dense

Capacity mode: Serverless

Model: multilingual-e5-large