I suspect that the experts at Pinecone know enough about RAG performance issues (both quality and speed) and chunking to be able to write a program or develop a web interface to optimize chunking of data into Pinecone, something like:
- User uploads files into a bucket/lake/folder that they want chunked.
- User chooses from a menu dialog about use intensions, that would solicit things like completeness vs discrimination (field of view vs focus), speed vs thoughtfulness, amount of surrounding context, etc.
- The app analyzes the files, considers the user’s intent, and chunks the data in a near optimal manner for Pinecone to achive that intent .
I’ll state right now, given my company’s burgeoning (and my personal) interest in RAG LLM AI, that I’d pay ~$300 for such an application or ~$10/Gig of content (txt, pdf, Excel, .docx, .ppt, JSON at least) for a web service.