I am on the free plan, with a starter index created in the us-west4-gcp-free region. I upserted thousands of vectors. However, after doing hundreds of upsert operations, I only see 26 vectors in my index’s vector count in the Pinecone UI. Is this because of rate limiting? I got no errors, and the upsertedCount response variable is correct.
This is my code:
I am calling upsertVectors right after getting the embeddings array for each chunk.
export const upsertVectors = async (filePath, chunks, embeddingsArrays, index) => {
const batchSize = 100;
let batch = [];
for (let i = 0; i < chunks.length; i++) {
const chunk = chunks[i];
const vector = {
id: `${filePath}-${i}`,
values: embeddingsArrays[i],
metadata: {
...chunk.metadata,
loc: JSON.stringify(chunk.metadata.loc),
pageContent: chunk.pageContent,
filePath: filePath,
}
};
batch.push(vector);
if (batch.length === batchSize || i === chunks.length - 1) {
try {
const result = await index.upsert({
upsertRequest: {
vectors: batch,
}
});
console.log(`upsert ${result.upsertedCount} done , resetting batch`);
} catch (err) {
console.log("error upserting vectors", err);
}
// reset batch
batch = [];
}
}
}
Thanks for your help!