Metadata size exceeds the limit - but I have not included any metadata in my index yet

General
1

I am getting an ApiException error indicating metadata size is too large, but I have not created or added any metadata to the index.

Running this line of code:

docsearch = Pinecone.from_documents(docs, embeddings, index_name=index_name)

is generating this error:

ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'date': 'Wed, 10 May 2023 17:02:04 GMT', 'x-envoy-upstream-service-time': '0', 'content-length': '115', 'server': 'envoy'})
HTTP response body: {"code":3,"message":"metadata size is 64839 bytes, which exceeds the limit of 40960 bytes per vector","details":[]}
2

+1 same here!

“code”:3,“message”:“metadata size is 155002 bytes, which exceeds the limit of 40960 bytes per vector”,“details”:}

3

this one I think I figured out, it wasn’t the size of the metadata per se, it was the size of the vectors I was sending per call that gives that error.

4

Can you let me know what you changed to fix it?

5

yeah my batch size. i was reading in a giant text file and breaking it into chunks. And it was working fine. Then I made the chunks way too big and got that error. So I went back to the first size chunks.

6

Out of curiousity, what embedding model are you all using? all-MiniLM-L6-v2?

7

I am using the OpenAIEmbeddings model.
What I don’t understand is that it seems this error would be thrown when I create the embeddings - not when I search the database. Of course, there is the distinct possibility that I am going about this all wrong and I don’t fully understand the process. Any guidance would be greatly appreciated.

8

If you’ve been able to embed your docs with OpenAI’s embedding and upsert them into the pinecone index, I would think you’ve got the plumbing looking good for the OpenAI embedding of your query. Can you post your upsert/query code (obviously removing your API keys!).

9

index_name = ‘fusion’
docsearch = Pinecone.from_documents(docs, embeddings, index_name=index_name)

10

Oh, you need to chunk your metadata. There are restrictions on a per-row level. Try making your document smaller or chunking it. I’m going to assume you’re using langchain rather than directly calling the pinecone APIs, right? Try a small document and see if you still get the error (as the error message indicates, try a doc with a size of less than 40k). Langchain is likely storing the document in the metadata field.

11

multi-qa-MiniLM-L6-cos-v1

12

I’m doing below, as I embed pretty big html tables parsed to text

for doc in documents:
    embed_vector = embeddings.embed_query(doc.page_content)
    embeds.append(embed_vector)
index.upsert(vectors=zip([doc.metadata['id'] for doc in documents], embeds, [doc.metadata for doc in documents]))

not calling embed_documents where I need to pass a fix chunk size

chunk_size = 1500
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
embeds = embeddings.embed_documents(texts, chunk_size)
index.upsert(vectors=zip(ids, embeds, metadata_list))

how do I avoid getting caught into chunk_size limit

13

@nsyed Based on the code you shared, it appears you are adding metadata with the third item in the tuple — [doc.metadata for doc in documents].

If you do not desire to have metadata on your records, try removing this from the upsert. If you want to attach metadata representing the text of the vectors, you will likely need to lower your chunk size.

14

Hi, im facing the same issue. I also split my docs to just 350 characters before loading them but getting still the error

15

For me it was a single entry that exceeded the size limit. I managed it by first checking the size of the text chunk and then split the text further using the splitter to get the text chunks. Here is the rough implementation:

EMBEDDING_CTX_LENGTH=8192
if cleaned_chum_size > 35*1024: # 35KB
    embeddings.chunk_size = int(int(EMBEDDING_CTX_LENGTH)/6)
    splitter_low = SemanticChunker(embeddings,  breakpoint_threshold_type="percentile")
chunks = splitter_low.split_text(cleaned_chum)

Here, I am using semantic chunking strategy. I feel it really gives good results but is a bit slower.