Pinecone Pod Storage + Vector max "token length"?

I’ve seen in the documentation under limits for Storage options: " Each p1 pod has enough capacity for 1M vectors with 768 dimensions."

I’m i correct in saying if i use a dimension length of 1536 (OpenAI embedding), that this would mean i can store 500k vectors. 768 * 2 = 1536 dims, 1M / 2 = 500k vectors.

Lastly, i wanted to find out if a vector has a “max length”. If i had a corpus of text in the range of 32k tokens (GPT4 32k model version or higher context length window - like MPT-7B [65K] ). Can a single vector in Pinecone store the full 32k tokens within the embeddings, if i wanted to do so - or do i have to do some text splitting magic on the text to make it fit into the “max length” of the vector. Just wanted to find out if vectors had a “max length”.

Hi @mndlovuca!

I’m i correct in saying if i use a dimension length of 1536 (OpenAI embedding), that this would mean i can store 500k vectors. 768 * 2 = 1536 dims, 1M / 2 = 500k vectors.

If I’m understanding your question correctly, no, that is not right. If you are using vectors of 1536 dimensions, you must create an index that is also 1536 dimensions.

Lastly, i wanted to find out if a vector has a “max length”.

Yes, there is a max size. Vectors can be a maximum of 20 MB. You can read more about our limits on our site.

Due to ‘lost in the middle’ problems (in addition to context window limitations, as you note), it’s unadvisable to cram all 32k tokens into a single vector (even if the context window allows it). You don’t want chunks of data that are too long, or else your vectorizer will lose important context. Conversely, you don’t want your chunks to be too short, or else you fail to include ENOUGH semantic data.

I hope this makes sense! Feel free to ask any followup Qs!

1 Like