I love the simplicity of using Pinecone, but can’t test it for my use case at the moment because of the 40kb metadata size limit. I am handling slightly larger documents that cant be chunked without losing context. Is there a reason why we can’t store more data in each vector? Just 100kb would be enough.
Hi @kristian! Glad you are liking the simplicity of Pinecone
The 40kb metadata size limit applies only to the metadata
field (which is optional) in the record. If you want to embed your documents without chunking them, you should be able to do so without issue. See Generate embeddings or our Quickstart for more info.
Let me know if you’re able to start testing or if you need additional support!
Hi @lauren.s ! I’m using Flowise for upserting my documents and I don’t think there’s a way to upsert text as anything other than metadata there. I don’t have a lot of documents so I could edit them in the record manager in Pinecone manually, but I don’t see a way to add data as anything other than metadata there.
I have seen older posts in the support forum where some users got an increase on their index’s metadata limit, is that a possible solution?
Hi @kristian, from my understanding, we can’t change the metadata limit but I can submit a feature request for you if you want to DM me your organization name.
Alternatively, have you considered using Pinecone Assistant to test?
@lauren.s Trying to DM but I can’t find a message function anywhere?
@kristian if you scroll to the very bottom of the left side nav, there’s a Messages > Inbox tab. You can send a message from there!
I really can’t find it, I don’t think I have it