How do Vector databases differ from ElasticSearch/Opersearch?
With the introduction of vector seasrch in ES and OpenSearch, those can be used as vector databases as well. There are differences between Pinecone and them (eg, filtering, real-time index updates) mainly due to the fact that Pinecone was designed as a vector database from the ground up.
Hi James, would you be able to discuss a bit more about Pinecone indexing and how it reindexes when we have a large volume of data?
Also, how can we scale the infrastructure requirements as the number of documents increases?
Hi sure, indexing is done effectively instantly. If you upsert 1M new vectors in a 1B index, those new vectors will be available to your queries within milliseconds. Reindexing is done periodically as the index changes.
At the moment we can help you switch to larger indexes if needed, in the future we plan on supporting the function to increase index size with new pods without the need to create a new index.