Wrong search results come back as high as the correct ones!

We are trying to create a simple semantic search engine following all the main instructions across a variety of places.

Here are our steps.

Take 40 images of a variety of dogs, houses, large buildings (like cathedrals).
Generate text descriptions and tags containing lots of relevant words and descriptions
Generate Embeddings of these descriptions (we have tried both Open Ai and Vertex Text Gecko)
Added the embeddings to a Pinecone index

Taken the search term ‘Dog’
Generated the equivalent embedding (so Openai - Openai or Vertex - Vertex)
Ran a search on Pinecone

Searching Dog brings back some houses as high as some dogs (the term dog is highly used in the dog description and not used at all in the houses)
Searching Animal brings back more houses (Using animal as a semantic search example - word is not used in any description)
Searching Pet also brings back more houses (Using pet as a semantic search example - word is not used in any description)

We are seeing results at 0.75 for each of Dog and Houses (I can share a full list)

What we expected to see was Dog to be about 0.9 for the term Dog - possible as low as 0.8 for Animal and Pet, but house to be significantly less - around 0.5 or lower

We are clearly doing something wrong in the process, but we have tried all sorts of models, index type etc and its roughly the same.

Can someone help to take a look?