The docs for the API indicate that the Log Sequence Number can be obtained from the headers of the response: Check data freshness - Pinecone Docs
However, the Python SDK (GitHub - pinecone-io/pinecone-python-client: The Pinecone Python client) does not expose this, at the very least not in the asyncio part of the SDK, and the alternative approach seems incredibly unideal as you’d essentially have to check the count of an entire namespace to compare against our expected record count (which has other drawbacks). Is there a way to get this header, both for upserts and for describe-index-stats, via the Python SDK? Or would the recommendation be to simply build the request by hand instead, and skip using the SDK entirely?
Welcome to the Pinecone forum, and apologies for the delayed response.
There’s currently no way to get that header in SDK responses, but I’ll share this ask with the team.
Can you share more about your scenario? It’s important to remember that Pinecone is an eventually consistent database, but we are making improvements with freshness speeds all the time. In fact, we are rolling out an optimized serverless architecture starting this week that will soon improve freshness noticeably. Eventually, the new architecture may enable strongly consistent reads. If you’re interested, you can read more about the new architecture here and here.
I am facing a similar problem with the .NET SDK. Is there any updates regarding implementing this in the SDK response? Our system needs a proper check of freshness, for now we are forced to query to check if the document id is available, and even then you are not guaranteed that all chunks have been upserted into the database.
@matthijs For what it’s worth, we ended up just using the API directly for this one endpoint (and the other endpoint to check what the current freshness is for polling afterwards) so that we could get the header response, rather than the SDK, and that pretty much solved all our problems. Obviously it’d be ideal to have it in the SDK, but that approach was just immediately far more effective than the workarounds we were trying, such as exactly what you’re trying (we ran into the same problem with that approach).
Sorry for my delay, @matthijs, and thanks for sharing your workaround, @ajohnston. There’s no change on this as of yet, but I’ll definitely pass on your feedback.