The dimensions for the open ai text-embedding-small/large models have changed in the wizard for creating indexes.
Large is now showing as 1024 isn’t of 3072. I now weirdly have some historical indexes that are 3072 and those created today are 1024.
Can someone help?
Hi @rich1, thanks for your post. text-embedding-3-large accepts 3072, 1024, or 256 dimensions. I confirmed on OpenAI’s site that 1024 is a valid dimensionality for text-embedding-3-large . Pinecone does support 3072-dimensionality indexes, but it takes up more storage, so it is generally not the default option.
If you want to create a 3072-dimension index, you can enter that into the Dimensions field on the index creation page:
Thanks Veronica, I know that I can set them manually I was just surprised the default changed as that isn’t the default dimensions of text-embedding-large. If you were to call text-embedding-large with default parameters and try to upload against your default settings it would fail. Ideally the default’s should match and they did last week, someone changed it I think?
Thanks for the additional context and feedback. I’ve raised this to the team to improve the UI experience. What are your thoughts on having a dropdown for all 3 dimensionality options, with 3072 as the default?
I noticed the same issue today which brought me to the community. Thanks, @rich1, for the post. The suggested dropdown for all 3 dimensions works for me.
@jayozer appreciate the feedback! I’ve passed it along to the team.