I try to implement the code released here : https://www.pinecone.io/learn/series/nlp/gpl/
Code :
import random
batch_size = 16
triplets =
for i in tqdm(range(0, len(pairs), batch_size)):
# embed queries and query pinecone in batches to minimize network latency
i_end = min(i+batch_size, len(pairs))
queries = [pair[0] for pair in pairs[i:i_end]]
pos_passages = [pair[1] for pair in pairs[i:i_end]]
# create query embeddings
query_embs = model.encode(queries, convert_to_tensor=True, show_progress_bar=False)
# search for top_k most similar passages
res = index.query(query_embs.tolist(), top_k=10)
# iterate through queries and find negatives
for query, pos_passage, query_res in zip(queries, pos_passages, res[‘results’]):
top_results = query_res[‘matches’]
# shuffle results so they are in random order
random.shuffle(top_results)
for hit in top_results:
neg_passage = pairs[int(hit[‘id’])][1]
# check that we’re not just returning the positive passage
if neg_passage != pos_passage:
# if not we can add this to our (Q, P+, P-) triplets
triplets.append(query+‘\t’+pos_passage+‘\t’+neg_passage)
break
with open(‘data/triplets.tsv’, ‘w’, encoding=‘utf-8’) as fp:
fp.write(‘\n’.join(triplets))
Error :
—> 38 res = index.query(query_embs_list, top_k=10)
39
40 # iterate through queries and find negatives
10 frames
/usr/local/lib/python3.10/dist-packages/pinecone/core/client/rest.py in request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
228 raise ServiceException(http_resp=r)
229
→ 230 raise ApiException(http_resp=r)
231
232 return r
ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({‘content-type’: ‘application/json’, ‘date’: ‘Mon, 24 Jul 2023 13:45:47 GMT’, ‘x-envoy-upstream-service-time’: ‘2’, ‘content-length’: ‘110’, ‘server’: ‘envoy’})
HTTP response body: {“code”:3,“message”:“Query vector dimension 12288 does not match the dimension of the index 768”,“details”:}
Tks a lot for any suggestion
best regards