# Removing duplicate embeddings

**URL:** <https://community.pinecone.io/t/removing-duplicate-embeddings/1186>\
**Category:** Support\
**Tags:** vector-database\
**Created:** [April 13, 2023, 10:20pm UTC](https://community.pinecone.io/t/removing-duplicate-embeddings/1186 "2023-04-13T22:20:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![joeyanuff](https://sea2.discourse-cdn.com/flex020/user_avatar/community.pinecone.io/joeyanuff/32/256_2.png) [@joeyanuff](https://community.pinecone.io/u/joeyanuff)\
**Post date:** [April 13, 2023, 10:20pm UTC](https://community.pinecone.io/t/removing-duplicate-embeddings/1186/1 "2023-04-13T22:20:36Z")

</div>

Working on a ChatYourData project with Langchain and Next.js, and search results from my Pinecone index suggest that I’ve somehow uploaded duplicate sets of embeddings: all my results are returning in identical pairs.

It’s a small (\<1000 vector) db and easily enough deleted and reimported, but it has me wondering: is there perhaps some simple native method of deduping a vector db?

---

<div class="post-metadata">

**Author:** ![jamesbriggs](https://sea2.discourse-cdn.com/flex020/user_avatar/community.pinecone.io/jamesbriggs/32/52_2.png) [@jamesbriggs](https://community.pinecone.io/u/jamesbriggs)\
**Post date:** [April 18, 2023, 2:22am UTC](https://community.pinecone.io/t/removing-duplicate-embeddings/1186/2 "2023-04-18T02:22:38Z")

</div>

no native method, best way I can think of is to search with returned vectors, grab IDs of those with duplicates, and delete one — it’s fairly straightforward with a smaller db like yours but this would get hard to do for a big dataset. In that case, you’d probably need to add a search and filter mechanism, a field like `{"clean": 0}` that once you have retrieved (and removed duplicates for) you update the metadata to `{"clean": 1}` and then during your next query you include `filter={"clean": 0}`

---

<div class="post-metadata">

**Author:** ![joeyanuff](https://sea2.discourse-cdn.com/flex020/user_avatar/community.pinecone.io/joeyanuff/32/256_2.png) [@joeyanuff](https://community.pinecone.io/u/joeyanuff)\
**Post date:** [April 18, 2023, 6:28pm UTC](https://community.pinecone.io/t/removing-duplicate-embeddings/1186/3 "2023-04-18T18:28:59Z")

</div>

Ah-ha, very clever approach, appreciate that James! (I opted to fix my doubles by reinitializing my db, but it’s helpful to think through the filter operation as you describe, thanks.)