As of 5pm EST on March 3rd, this incident is resolved and all affected indexes on the Starter (free) tier have been restored. Read the complete incident report to learn what happened and what we’re doing as a result.
We’re sharing an update about an incident that occurred yesterday that impacted indexes on the Starter (free) plan.
On March 1 at 3:00pm EST, a script meant to delete inactive indexes in the Starter (free) tier was inadvertently run on certain active indexes on the Starter (free) plan.
Our engineering team immediately halted index operations to investigate the issue. By 4:30pm the same day, all operations returned to normal and some affected indexes were restored. Indexes and operations for users on paid plans (Standard, Enterprise, or Enterprise Dedicated) were not affected.
By 7pm an email was sent to all affected users with the information we knew at that time.
At the time of this posting, our engineers were able to recover some indexes already. We are still working on recovering as many of the rest as possible. We will update affected users directly and everyone else in this post as updates become available.
We know that data is the lifeblood for our users, and they trust Pinecone with that data. We also know our users put Pinecone at the core of their applications and trust us to keep running. This trust is foundational to our business. We know that incidents like this impact your business, and we take this extremely seriously. We treat every incident involving data with the utmost severity and urgency.
UPDATES
March 2, 4:40pm ET - We have recovered 125 indexes out of 515 that were deleted. We are continuing work to recover additional indexes and will share more updates here.
March 3, 1:20am ET - We have restored a total of 388 indexes out of 515 that were deleted. We are continuing work to recover additional indexes and will share more updates here.
The recovered indexes have the same name as before. Users who recreated their indexes manually with the same name will see the recovered index as [original-name]_recovered. Affected users who manually recreated their indexes and upgraded to a paid plan should decide which of the two indexes to keep, to avoid being billed for identical indexes.
March 3, 6:15pm ET - We have now restored all 515 of 515 affected indexes. A small number of restored indexes (19 of 515) are missing updates from up to 2 hours before the incident on March 1, 3pm EST. For those indexes, we are investigating the exact timestamp of the last update and will contact the affected users directly.
Now that the recovery effort is completed, we will begin working on a post-mortem report with next steps that we will share with you soon.
Your patience and understanding is greatly appreciated during this process.