Dear Support,
We are executing this code inserting 100 rows per time with batch process as per your recommendation in your web site.
Please tell us approximately how much time should we spend to insert these 100 vectors.
We tried storing 1000 vectors in pinecone, it took us 7 seconds and when we tried inserting 100 vectors, it took us 0.37 seconds.
I have tried this code.
start_time_embedding = time.time()
def chunks(iterable, batch_size=100):
“”“A helper function to break an iterable into chunks of size batch_size.”“”
it = iter(iterable)
chunk = tuple(itertools.islice(it, batch_size))
while chunk:
yield chunk
chunk = tuple(itertools.islice(it, batch_size))
vector_dim = 1536
vector_count = 100
example_data_generator = map(lambda i: (f’id-{i}', [random.random() for _ in range(vector_dim)]), range(vector_count))
async_results = [
index.upsert(vectors=ids_vectors_chunk, async_req=True)
for ids_vectors_chunk in chunks(example_data_generator, batch_size=100)
]
end_time_embedding = time.time()
elapsed_time_embedding = end_time_embedding - start_time_embedding
print(f"Conversational embedding time: {elapsed_time_embedding} seconds")