Offset / pagination for `documents/search` (and `/query`)

The new Documents API endpoints do not support offset or a pagination token like the Vector API endpoints do.

So the only way we can do pagination e.g. for page 2 is fetch 2x the amount and slice at the moment.

It seems like a bit of a regression and will cause us issues with the Drupal Search API/AI Search integration (I’m the primary maintainer of AI Search there + the AI VDB Provider Pinecone modules).

Issue in Drupal Gitlab queue here Client Challenge