Syncing data from warehouses into Pinecone

General
,
1

How are people syncing large amounts of data into Pinecone currently? Is it purely with custom scripts?

I see one of the YC startups this year (Sidekick) start doing a “Fivetran for LLMs” which takes data from SaaS apis like Notion, Zendesk, Slack, etc and syncs it to Pinecone/Weaviate.

However, it seems like most companies already keep a full representation of their data in a data warehouse (BigQuery/Snowflake) already using tools like Fivetran/Airbyte/Stitch/etc.

Is there a tool that helps sync from data warehouses into Pinecone?

1 Like
2

Have you found an answer to your question?

3

Hi @lizozom @ieoe, Have you found a good tool for this? I’m a startup founder working on exactly this, i.e. syncing large amounts of data from warehouses and blob storage into vector dbs. I’d love to learn more about your workflow, share my approach, and get your feedback. Would you be open to connecting?

4

There’s now an Airbyte connector for Pinecone! Introduction to Airbyte and the Pinecone connector | Pinecone