My upsert doesn't upsert

Hi there, when I upsert they upsert doesn’t get uploaded… Here is what I have done

import json
from pinecone import Pinecone

pc = Pinecone(api_key="my_api_key")
index = pc.Index("blabla-bot")
filename = 'output_vectors_chunk_1.json'

# Open the file and load vectors
with open(filename, 'r') as file:
    vectors = json.load(file)
    index.upsert(vectors=vectors)
    print(f"Upserted vectors from {filename}")

My json looks like this:

[
    {
        "id": "dfa273a7-0498-4726-bbec-57f57200e8c5-0",
        "values": [
            -0.03402634337544441,
            0.05481314659118652, # etc
        ],
        "metadata": {
            "video_title": "Title of the video",
            "video_url": "https://www.youtube.com/watch?v=video_id",
            "original_text": "blablablabla"
        }
    },
    {
        "id": "dfa273a7-0498-4726-bbec-57f57200e8c5-1",
        "values": [
            -0.033837657421827316,
            0.029345935210585594, # etc

This should be the correct format.

The index seems to be up and running, I checked with pc.list_indexes():

{'indexes': [{'dimension': 1536,
              'host': 'blabla-bot-25ieqpn.svc.gcp-starter.pinecone.io',
              'metric': 'cosine',
              'name': 'blabla-bot',
              'spec': {'pod': {'environment': 'gcp-starter',
                               'pod_type': 'starter',
                               'pods': 1,
                               'replicas': 1,
                               'shards': 1}},
              'status': {'ready': True, 'state': 'Ready'}}]}

I’m extremely lost, according to the docs I did everything right but somehow no vectors are uploaded :frowning:

The prints do get through to the code is running at least… but when I refresh my page still nothing there.

print(index.describe_index_stats())

Returns

{'dimension': 1536,
 'index_fullness': 0.0,
 'namespaces': {},
 'total_vector_count': 0}

Any help would be greatly appreciated

Hi @piet. Thanks for posting this question!

Did you receive an error that tells you that the vectors were not upserted, or are you just not seeing the vectors in the Pinecone console or the describe_index_stats response?

I just updated my original post, I don’t see them for both. Thanks for the response!

I received no error so it’s hard to know what went wrong

Ah, I see your updated post now. How long ago did you upsert the records, and how many records did you upsert? There can be a slight delay before describe_index_stats returns the accurate vector count, though it shouldn’t be long.

It was about 30 minutes ago. I just restarted everything and now I receive error:

pinecone.core.client.exceptions.ServiceException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({‘content-type’: ‘application/json’, ‘Content-Length’: ‘150’, ‘x-pinecone-request-latency-ms’: ‘4412’, ‘date’: ‘Wed, 07 Feb 2024 22:33:36 GMT’, ‘x-envoy-upstream-service-time’: ‘3178’, ‘server’: ‘envoy’, ‘Via’: ‘1.1 google’, ‘Alt-Svc’: ‘h3=“:443”; ma=2592000,h3-29=“:443”; ma=2592000’})
HTTP response body: {“code”:13,“message”:“We were unable to process your request. If the problem persists, please contact us at https://support.pinecone.io”,“details”:}

Maybe it’s a server error?

My apologies Jesse for this emotional rollercoaster. The error is gone, but after my upsert attempt still nothing is coming through. Maybe I should sleep on it and check tomorrow.

They came through! Just 10 rows though. Maybe that’s standard for a free plan?

No need to apologize, @piet! We expect upserts to be very fast, so I’m sorry this hasn’t been smoother for you. In any case, your records should be available to query in way less than 30 minutes. I’ll look into next steps. In the meantime, can you share how many records you upserted, and how many total records are in the index?

Actually, @piet, I think it’s best if you could open a support ticket at https://support.pinecone.io so our team can investigate.

Will do, thanks for your quick replies!

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.