Pinecone Assistant - chatStream topk and snippet_size

Support
1

Hi everyone,

I’m trying to limit how much context the Assistant sends to the LLM because the defaults (top_k = 16, snippet_size ~2000) are just too expensive for my use case. The documentation under “Control the context snippets sent to the LLM” says you can pass something like:

context_options: {
  snippet_size: 1000,
  top_k: 10
}

However, when I try it in the Node SDK (v6.0.0), my calls fail because context_options isn’t recognized:

const stream = await assistant.chat({
  messages,
  stream: true,
  context_options: { snippet_size: 1000, top_k: 7 }
});

MSG: Pinecone chatStream error PineconeArgumentError: Object contained invalid properties: stream, context_options. Valid properties include messages, model, filter, jsonResponse, includeHighlights.

I’m guessing the SDK types or request validator haven’t been updated yet. Has anyone figured out a clean way to override those defaults in the current SDK? Thanks!

2

Hello @idgonzabr and welcome to the forums!

You are correct. The context_options config is not in the latest TypeScript SDK. We want to correct this ASAP and should have a patch release soon. Meanwhile, you can use the API directly to test if the functionality meets your expectations. Alternatively, you can try to provide your implementation of the FetchAPI to the constructor, where you augment the JSON generated by the SDK with the desired context_options.

I hope this helps, and I’m sorry for the inconvenience!

3

Hi @idgonzabr

Thanks for your patience! I’m happy to report that a new version of the Node client (which now contains the ChatContextOptions) is out.

Please try it and let us know whether it meets your expectations.

4

Uff this is huge, many thanks @milen , I will try it.