Hi all! I am a fresh graduate working on an AI project in hopes it will help me find a job.
I am working on an LLM powered chatbot that uses RAG to help the user buy group insurance. My intention is to first filter plans based on eligibility information from the user, such as their business size and what state they are in.
Currently, insurance plan information is stored in a dense index in JSON format, with each plan having the same structured keys but unstructured values to capture the nuances of each plan. Each plan also has the same structured metadata keys with structured values (literals confined to few options). The metadata includes information such as business size, what states the plan is available, whether the plan has local or national coverage, etc.
I planned to use this metadata to perform a filtered vector search in order to narrow down the plans. However, I began to consider using a sparse index instead for certain fields, such as the states the plan is located in as this is more unique to each plan. I was wondering how metadata filtering and sparse searches are different under the hood when it comes to searching a Pinecone DB? I can use this knowledge to make a more informed decision on how to rearchitect my solution. Any other suggestions are also greatly appreciated!
Welcome to the Pinecone forums and congrats on graduating! As you work on your project, feel free to share your progress here or even on social – we’d love to see it.
Think of metadata filtering as narrowing down the results to only those that are valid for the query and a sparse index search as a keyword search. Sparse is going to give you both contextual understanding (when matching on the word “bull” in the query, does it mean “bull market” or “bull fight”?) and semantic weighting (results related to “bull market” are weighted higher than those like “bull fight”).
In your case, I think your original inclination to use metadata filtering makes the most sense as these really aren’t keywords but instead you want to narrow your results to those that are valid.
For instance, if they live in Minnesota, Wisconsin plans are not relevant. Without a metadata filter, you’ll end up searching the entire namespace and less relevant results may be returned alongside the more relevant results, like those plans from Wisconsin even if your user is looking for Minnesota plans.
For business size, it might depend on how that’s stored/how it’s being queried (keyword like “0-100 people” vs a variable term like “small business” or “small biz” or “SMB”) and even how those plans work (is a “small business” plan valid for anyone or only those who have a “small business”?). If it’s stored as a key like “0-100 people” or “Small business”, metadata filtering may be more appropriate. If the user’s query doesn’t exactly match (e.g. “small biz” or “50 employees”), sparse will not be as helpful here.
For local/national coverage, I’d ask the same questions as above. Are you narrowing your results to only those valid or are you searching on a keyword and need the context and weighting?
Another question to ask yourself is whether you’re doing the filtering programmatically based on the user’s profile/account info or will it be part of the user’s query “I’m looking for local plans in MN for my small biz.”?
Hope that’s helpful! Here are a couple articles that talk more about each of these: