API rate limiting?

I started off using the API fine, inserting in batches of 30 with

for ids_vectors_chunk in chunks(yield_insert_tuples(ml_cxr_datalake), batch_size=10):

Made it to about 200K vectors but very slowly, full dataset is around 400K so I wanted to speed that up and tried

with pinecone.Index(<vector-index>, pool_threads=30) as index:
    # Send requests in parallel
    async_results = [
        index.upsert(vectors=ids_vectors_chunk, async_req=True)
        for ids_vectors_chunk in chunks(
            yield_insert_tuples(ml_cxr_datalake), batch_size=30
    # Wait for and retrieve responses (this raises in case of error)
    [async_result.get() for async_result in async_results]

This stopped inserting after the first batch, I went back to try the first method and again only managed to insert 30, as if I’m being rate limited. This was on the free account in order to evaluate the tool for further use against open-source options like Weaviate and Milvus. I’ve now switched to the Standard paid account to see if it would resolve the probelm but I’m getting similar behaviour.

Very confident that the small custom part of the code (retrieving the metadata) is not responsible for the problem but would welcome any suggestions wrt how I should proceed. Some of that confidence comes from the fact that the initial batch inserts fine each time and only freezes on the second, the batching code has come from your docs.

def chunks(iterable, batch_size=1000):
    """A helper function to break an iterable into chunks of size batch_size."""
    it = iter(iterable)
    chunk = tuple(itertools.islice(it, batch_size))
    while chunk:
        yield chunk

Hi @GeorgePearse
Do you see any errors when the upserts stop or are they just timeouts ? We do have a clear error message for when your index is getting rate limited so it wouldn’t be ambiguous.
Also, I would suggest giving gRPC index a try, you should see better performance.