Say that I’m indexing and embedding thousands of books: Should I store all these books in one index under different namespaces? When is it recommended to create new indexes? Should I do this when the objects embedded/embedding sizes change?
Thanks!
Have a look at: Multitenancy
If you ever want to search across books, then put them all in one namespace and use metadata to distinguish between books.
If you never need to search across books, then put them in namespaces to reduce query latencies (you’ll have a smaller search space).
The only reason to have separate indexes is if you’re using a different model for each book, since an index can only have one fixed dimensionality (eg 1536-dim embeddings or 768-dim embeddings, but not both). This is uncommon.