pinecone-datasets might be more oriented towards Pinecone public specific, but was hoping to use as a consistent storage format + to make upsert easier.
I have pyarrow 15.0.0 installed (for other reasons in working venv). But it is looking like the pinecone-datasets requires pyarrow<12.0.0 and >=11.0.0. Reason for that specific Pyarrow version? Could that be tested and updated in a future pinecone-datasets release?
I seem to have a related issue. When I try to install pinecone-datasets, I’m having an issue with building the pyarrow wheels. I can’t quite figure it out, but it seems like there is something incompatible between my visual studio version (2022), the cmake configuration and pyarrow. I tried to install using binaries to avoid the cmake/visual studio issue, but can’t find binaries for such an old version of pyarrow.