Total vector count not increasing

Support
1

I’m trying to upsert in a loop like so:

for topic in topics:
    newsapi = NewsApiClient(api_key='API_KEY')


    all_articles = newsapi.get_everything(
                                        q=topic,
                                        from_param='2024-08-01',
                                        to='2024-08-31',
                                        language='en',
                                        sort_by='relevancy',
                                        )
   
    article_content = []



    for i in all_articles['articles']:
        article_content.append(str(i['description']))


    print(len(article_content))


    embeddings = generate_embeddings(article_content)
    dense_embeddings = generate_dense(article_content)


   
    vectors = []
   



    for embedding in range(len(embeddings)):
        vectors.append(
            {
                "id" : "article_" + str(embedding),
                "values" : dense_embeddings[embedding],
                "sparse_values" : embeddings[embedding],
                "metadata" : {"text": article_content[embedding]}
            }
        )
    index.upsert(vectors)


    print(index.describe_index_stats())

Each run of the loop should upsert 100 vectors. However, when I print vector stats, it only upserts on the second run of the loop and in no other run. It looks like this:

100
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 2798.16it/s] 
{'dimension': 768,
 'index_fullness': 0.0,
 'namespaces': {},
 'total_vector_count': 0}
100
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 3074.57it/s] 
{'dimension': 768,
 'index_fullness': 0.0,
 'namespaces': {'': {'vector_count': 100}},
 'total_vector_count': 100}
100
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 4996.61it/s] 
{'dimension': 768,
 'index_fullness': 0.0,
 'namespaces': {'': {'vector_count': 100}},
 'total_vector_count': 100}
100
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 4062.12it/s] 
{'dimension': 768,
 'index_fullness': 0.0,
 'namespaces': {'': {'vector_count': 100}},
 'total_vector_count': 100}

Is there any way to fix this?

2

Never mind it is working now. I think the issue was that the id was not changing throughout the runs. I changed that and now it’s working.

1 Like
3

@tvishachoubey,

Roger - thank you for returning to post a solution and thanks for using the forums!

Best,
Zack