Collection on Starter Plan stuck 'Initializing'

Dear Support team,
on my Free tier project, there seems to be anomalies happening in the Collections.
I tried to create ‘books-ada-mvp-metadatatext-jan11’ from the index ‘books-ada-mvp-metadatatext’ and it has been in ‘Initializing’ mode for several hours. Then I tried to create another collection ‘books-ada-mvp-metadatatext-january’, which got stuck to in the same status. Theoretically, this 2nd collection shouldn’t even have been allowed/happened in a Free tier project as the max is 1, right…?

Also, I had a question whether a Collection stored in the Free tier can be used/sourced for an index creation on the Standard plan/different project. I’ve upgraded my account today so I’m hoping I can take what I had on my Free tier index and continue building up on a new Standard index.

In any case, I thank you in advance for addressing this. Much appreciated.

Hey @regutonlabs. Were you creating the collections through the console or the API directly?

Because indexes and collections are in separate cloud regions it’s not possible to create an index in a different project than the source collection. But it looks like you only have ~110k vectors in your index, so I would recommend copying from your current index into a new one in a new project and either updating the original index with a metadata key marking vectors as moved or deleting them from the original index and then restoring the index from a collection if you still need it in the free project.

The order of operations would be:

  • Initialize a connection to Pinecone in your current free project.
  • Run a query with an example vector (could be anything; [0.1] * dimensions_in_your_index works well for this) with a top_k of 1000.
  • Initialize a new connection to Pinecone in the target project.
  • Upsert the vectors returned by the query above.
  • Initialize a connection back on the free project again and mark/delete the vectors that were returned, using their vector IDs to do so.
  • Rinse and repeat.

Let me now if you’d like code examples for those operations.

1 Like

Hey @Cory_Pinecone - thanks for your reply! Collections were created through the console… The two I was referring to still appear as ‘initializing’. So, as of now I’ve not been able to get an up-to-date collection for my free-tier. However now I see an old previous Collection called ‘books-ada-mvp-metadatatext-19oct’ which I thought I had removed.
All that said, it sounds like your approach is probably what I’ve got to do if I want to migrate my vectors from the free tier index into the paid one.
Please kindly share some code examples, as I follow everything you mention except the 2nd step which I’m not too clear on what you mean exactly…
Thanks for all your help and support!

1 Like

Hey @Cory_Pinecone,
I’ve been trying to write the steps you suggested but i’m encountering errors because of the class format in which the query matches are retrieved. Please kindly suggest whether there’s something I’m not thinking right. Certainly, you can’t simply upsert the vector matches in the query because they each come as class ‘pinecone.core.client.model.scored_vector.ScoredVector’ - so how to convert?
See my code below and currrent error:

def migrate_free_tier_to_serverless():
    from pinecone import Pinecone, ServerlessSpec
    index_name = "my index"
    # Initialize a connection to Pinecone in your current free project.
    free_pc = Pinecone(api_key="myapi", environment="myenv")
    free_index = free_pc.Index(index_name)
    print(free_index.describe_index_stats())
    # Run a query with an example vector (could be anything; [0.1] * dimensions_in_your_index works well for this) with a top_k of 1000.
    origin_query = free_index.query(
        vector=[0.1] * 1536,
        top_k=3,
        include_metadata=True,
        include_values=True
    )
    # Initialize a new connection to Pinecone in the target project.
    serverless_pc = Pinecone(api_key="myapi")
    serverless_index = serverless_pc.Index(index_name)
    # Upsert the vectors returned by the query above.
    unscored_query_matches = []
    for query_match in origin_query.matches:
        unscored_query_match = {
            "id": query_match.id,
            "metadata": query_match.metadata,
            "values": query_match.values
        }
        unscored_query_matches.append(unscored_query_match)

        serverless_index.upsert(unscored_query_match)
    # Initialize a connection back on the free project again and mark/delete the vectors that were returned, using their vector IDs to do so.

    print(serverless_index.describe_index_stats())
migrate_free_tier_to_serverless()

ERROR:

File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pinecone/utils/error_handling.py”, line 10, in inner_func
return func(*args, **kwargs)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pinecone/data/index.py”, line 161, in upsert
return self._upsert_batch(vectors, namespace, _check_type, **kwargs)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pinecone/data/index.py”, line 184, in _upsert_batch
vectors=list(map(vec_builder, vectors)),
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pinecone/data/index.py”, line 180, in
vec_builder = lambda v: VectorFactory.build(v, check_type=_check_type)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pinecone/data/vector_factory.py”, line 60, in build
raise ValueError(f"Invalid vector value passed: cannot interpret type {type(item)}")
ValueError: Invalid vector value passed: cannot interpret type <class ‘str’>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.