Table Question Answering - multi-table answers, table size limits, etc

General
1

Trying to solve a problem, and Pinecone looks interesting, but think I may be going down the wrong path.

I was hoping to take some data like this - https://data.cms.gov/provider-data/dataset/mj5m-pzi6 - and ask English questions on it. Maybe something like ‘which internal medicine doctors went to the University of Kentucky College of Medicine who work in Virginia’.

Trying to understand whether Pinecone is something that could help - looking at the table question answering topic, even though that uses as an example 20000 small tables from wikipedia, as opposed to 1-2 large tables like these (million row+ tables).

Just looking to see if this is worth pursuing or not or just not a problem Pinecone is trying to solve - since again, the examples are sort of what I’d want to support, even though I may also need to answer questions which span multiple tables. (like, instead of Virginia, in cities > 500,000 people, so I’d need census information as well).

2

i would make it a postgres database , and use natural language to query it.
it saves $$ as you don’t need to embed the text
in other words, you use openai only to translate your natural language into a postgres query

3

there is another option