Documentation on fetch the record IDs based on their common ID prefix is incorrect

[alex6]

Due to the fact that Serverless doesn’t support deletion of record with metadata filtering, the doc here explains how to use the list operation to fetch the IDs of the records and then delete the records by ID.
The doc suggests that “If there are more than 100 records to return, the list response contains a pagination key”:

# Response:
# {
#   "vectors": 
#     [
#       { "id": "doc1#chunk1" },
#       { "id": "doc1#chunk2" },
#       { "id": "doc1#chunk3" },
#       { "id": "doc1#chunk4" },
		...
#     ],
#   "pagination": {
#     "next": "c2Vjb25kY2FsbA=="
#   }
# }

However in reality I am getting a pagination key with much less than 100 vectors in the response:

pagination: {
 next: 'eyJza2lwX3Bhc3QiOiI2MGZmODg5ZS1hYWY1LTRiNTktOWUwNS1mNmRmNTVhOWM5Yjc6Y2h1bmtfMyIsInByZWZpeCI6IjYwZmY4ODllLWFhZjUtNGI1OS05ZTA1LWY2ZGY1NWE5YzliNyJ9'
}

Decoding the pagination value reveals the following in my case:

{
  "skip_past": "60ff889e-aaf5-4b59-9e05-f6df55a9c9b7:chunk_3",
  "prefix": "60ff889e-aaf5-4b59-9e05-f6df55a9c9b7"
}

which is OK but not documented. What should be the decoded value of the pagination key if there are more than 100 records? skip_next, skip_forward?

[zeke]

Hi @alex6, thanks for your post!

I’ve connected with our product and documentation teams and can share the following about the current expected behavior of the List operation. Please keep in mind that we will continue striving to improve the performance and usability throughout the public preview period for Pinecone serverless.

• Returning a pagination token when the next page is empty is a known issue that we will hopefully have a fix out for soon.
• At present, users should not assume that a page containing fewer results than the limit value is necessarily the last one. Instead, they should always check the pagination.next field. Once we implement the fix, this field will be empty if no more pages exist.

We have an open PR with documentation updates to clarify the expected behavior.