Record count mismatch the number of upserted vectors

Hi,

So I was trying to upsert a new batch of embeddings to our serverless index, and the number of vectors that I upserted is 6545 (and I am very sure these new vectors are unique and not overlapped with what existed in the index), and our log shows that they were indeed all upserted successfully and there is no error while upserting. Before this upsert operation, the number of record counts in our index was 4829, and after the upsert it should become 6545+4829 = 11374, however, in both the console and describe_index_stats() it all shows 11370, and I went to look up the Metrics in the console and look at the record count chart with time passed - right before it settles on 11370, the record count in the chart was shoot up to 11434 (which is more than the total vectors we have before plus the upserted!), then quickly reduced to 11412.667, then finally settled on 11370, but it should have been 11374, missing 4 vectors…

I did wait and checked again later (checked the data freshness guide, and aware of the small delay), but it is still 11370. For previous operations, the upsert job has always been accurate, getting to the right record count after a few seconds after the operation is done, but I have no idea why we are missing 4 vectors this time. Could anyone help explain?

Thanks

I just saw this happening to me too… DB shows 410 records, I upserted these, and then I see 419 records, then 451. I waited like 30 minutes or an hour, and then I saw the record count go down again.

Editing 11 minutes after: The record count just updated back to 410 records.

@mike24dzy and @jorge.jimenez,

Thank you for raising this point. I got a few follow-up questions:

  • When did you see this issue?
  • Can you reproduce it regularly
  • Can you provide the index URL? Feel free to private message it to me

I saw this issue today.

Yes, all I have to do is upsert, and I see the interactions. I have screenshots and everything, can you send me a private message? I can share the screenshots as well as the index url.