I started off using the API fine, inserting in batches of 30 with
for ids_vectors_chunk in chunks(yield_insert_tuples(ml_cxr_datalake), batch_size=10):
index.upsert(vectors=ids_vectors_chunk)
Made it to about 200K vectors but very slowly, full dataset is around 400K so I wanted to speed that up and tried
with pinecone.Index(<vector-index>, pool_threads=30) as index:
# Send requests in parallel
async_results = [
index.upsert(vectors=ids_vectors_chunk, async_req=True)
for ids_vectors_chunk in chunks(
yield_insert_tuples(ml_cxr_datalake), batch_size=30
)
]
# Wait for and retrieve responses (this raises in case of error)
[async_result.get() for async_result in async_results]
This stopped inserting after the first batch, I went back to try the first method and again only managed to insert 30, as if I’m being rate limited. This was on the free account in order to evaluate the tool for further use against open-source options like Weaviate and Milvus. I’ve now switched to the Standard paid account to see if it would resolve the probelm but I’m getting similar behaviour.
Very confident that the small custom part of the code (retrieving the metadata) is not responsible for the problem but would welcome any suggestions wrt how I should proceed. Some of that confidence comes from the fact that the initial batch inserts fine each time and only freezes on the second, the batching code has come from your docs.
def chunks(iterable, batch_size=1000):
"""A helper function to break an iterable into chunks of size batch_size."""
it = iter(iterable)
chunk = tuple(itertools.islice(it, batch_size))
while chunk:
yield chunk