We developed the Apify Pinecone integration, details here. Recently, our users have reported issues with “too many requests” exceptions—see this issue report. As a quick fix, we implemented a retry mechanism, but we’re still encountering the issue.
I tried replicating it and received the same exception. However, when I checked the Pinecone index monitoring, the number of requests appeared to be within acceptable limits.
We are using the Pinecone integration in LangChain.
I recognize that our current solution is not fully efficient. Every time we perform a crawl, we compare the newly crawled data with the last crawl in the Pinecone database. This comparison process is not optimized for Pinecone specifically (since we use similar code for other databases), and it’s likely contributing to the 429 exceptions. However, based on the metrics in the monitoring tab, this shouldn’t be causing the issue.
Or is my understanding of the metrics incorrect?
I would appreciate any help.