Hi,
I am new to using Pinecone but I am using it to store sentence embeddings along with some metadata including the original text for a similarity search use case.
I am trying to store my embeddings along with the metadata in batches of 100 but I am running into the following error:
...
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'date': 'Fri, 03 Jun 2022 17:48:20 GMT', 'x-envoy-upstream-service-time': '4', 'content-length': '113', 'server': 'envoy'})
HTTP response body: {"code":3,"message":"metadata size is 7271 bytes, which exceeds the limit of 5120 bytes per vector","details":[]}
It looks like there’s a limit to metadata of 5MB so I’m not sure why I’m running into the limit here. I assumed the 5MB limit was for each vectors but is the 5MB shared across all vectors?
1 Like
We are actually increasing the metadata limit from 5K to 10K as of today’s release. This should resolve the issue you are seeing. This will mean 10K for each vector for the second part of your question.
We are also introducing the concept of selective metadata indexing. This allows you to choose which metadata fields are indexed and which are stored. Indexed metadata is filterable and can be used at query time to limit the results. Stored fields are retrievable but not filterable so they can be brought back in the results for your application but they cannot be used to limit the results. The purpose of this feature is to minimize the overall footprint of your index while maintaining the usefulness of the metadata.
See: https://www.pinecone.io/docs/manage-indexes/#selective-metadata-indexing
Pinecone
1 Like
What’s the workaround when our metadata is more than 10k? Any documentation about it? Should we chunk into multiple documents, etc;?
There isn’t a single workaround as it depends on your use case. What kind of metadata are you storing? Are there a lot of key/value pairs, or is it a small number (or just one) that is exceptionally large?
Just a single metadata is big, imagine a document like Google Doc / Notion, etc.
Any other workaround than chunks?
On the other hand, the advantage would be able to search for specific part of the document; but I’d have preferred the basic feature to work with large doc first
The two best options are either to chunk the source document up and store multiple vectors of it, one per chunk, with the text limited to < 10k per vector (the max size for metadata). Or, and this is the one that most of our customers use, store the document in a bucket and only keep the URL to that storage in metadata. It depends on what you need to return by your searches; if you have to return the entire document or just the relevant section of it. Even in the case of storing the documents in chunks, you may want to include a URL to the entire document.
Pinecone is heavily optimized for search and retrieval for vectors. It’s not intended to be a general-purpose database, so best practices suggest not treating it as one. Storing too much metadata can have an adverse effect on the performance of searches and reduce how many vectors you can store in each index.
We have a new feature being developed that will change how metadata is stored, which should be coming out later this year. That feature will mean storing an entire document as metadata would be more viable. But today, it’s not a recommended practice.