Bug Report
Title: Issue with $and Operator in Metadata Filter for assistant.chat_completions
Description:
I encountered an issue when using the $and operator in the metadata filter for assistant.chat_completions. The documentation (Four features of the Assistant API you aren't using - but should | Pinecone) suggests this is a valid use case, but the API call returns a 400 Client Error.
Steps to Reproduce:
- Set up a metadata filter using the $and operator as follows:
metadata_filter = {
“$and”: [
{“date”: {“$gte”: 1698969600}}, # Unix timestamp for 2023-11-03
{“date”: {“$lte”: 1699142400}} # Unix timestamp for 2023-11-05
]
}
response = assistant.chat_completions(
messages=chat_context,
stream=True,
filter=metadata_filter
)
- Call assistant.chat_completions with the filter.
Expected Behavior:
The API should return the filtered chat completions matching the metadata criteria.
Actual Behavior:
The API raises a 400 Client Error with the following traceback:
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url:
…/assistant/chat/test/chat/completions*
…
ValueError: Error in chat completions streaming: 400 Client Error: Bad Request for url: …/assistant/chat/test/chat/completions*
Additional Information:
• The files with relevant metadata are correctly uploaded and available:
[FileModel(name=‘file-4.txt’, id=‘66228d1a-ac09-408c-be81-bb39cee82b0f’, metadata={‘date’: 1699056000.0, ‘source’: ‘test/file-4.txt’}, …),
FileModel(name=‘file-5.txt’, id=‘a294006d-8d5f-45ff-ab90-9873467a4ae9’, metadata={‘date’: 1699142400.0, ‘source’: ‘test/file-5.txt’}, …),
FileModel(name=‘file-3.txt’, id=‘c6d652d8-a356-4d10-b1bb-5808b8b7cad5’, metadata={‘date’: 1698969600.0, ‘source’: ‘test/file-3.txt’}, …)]
• Environment:
• Python version: 3.11
• Pinecone: 5.4.2, Pinecone-plugin-assistant: 0.4.3
• The error consistently occurs with the above filter.
Suggested Debugging Steps:
-
Verify if the $and operator is supported for assistant.chat_completions filters.
-
Check if the filter’s syntax adheres to the API’s expected format.
-
Investigate the 400 error response to provide additional context about the root cause.
Attachments:
Full traceback and relevant code snippets are included above for reference.
Please advise on how to resolve this issue or whether there are alternative approaches to achieve the desired filtering.
Best regards,
Jim
from pinecone import Pinecone
from pinecone_plugins.assistant.models.chat import Message
import datetime
pc = Pinecone()
assistant = pc.assistant.Assistant(assistant_name="test")
# files_info = [
# {"file_path": "test/file-1.txt", "date": "2023-11-01"},
# {"file_path": "test/file-2.txt", "date": "2023-11-02"},
# {"file_path": "test/file-3.txt", "date": "2023-11-03"},
# {"file_path": "test/file-4.txt", "date": "2023-11-04"},
# {"file_path": "test/file-5.txt", "date": "2023-11-05"}
# ]
#
# # We'll convert the date to a Unix timestamp
# for file_info in files_info:
# file_info["timestamp"] = int(datetime.datetime.strptime(file_info["date"], "%Y-%m-%d").timestamp())
#
# # Upload the files
# for file_info in files_info:
# response = assistant.upload_file(
# file_path=file_info["file_path"],
# timeout=None,
# metadata={"source": file_info["file_path"], "date": file_info["timestamp"]}
# )
# # responses.append(response)
# Then, we can filter the response based on the date associated with the file
metadata_filter = {
"$and": [
{"date": {"$gte": 1698969600}}, # Unix timestamp for 2023-11-03
{"date": {"$lte": 1699142400}} # Unix timestamp for 2023-11-05
]
}
# This API filter works
files = assistant.list_files(filter=metadata_filter)
print(files)
# This returns ValueError: Error in chat completions streaming: 400 Client Error: Bad Request for url: ...assistant/chat/test/chat/completions
chat_context = [Message(role="user", content="What is the maximum height of a red pine?")]
response = assistant.chat_completions(
messages=chat_context,
stream=True,
filter=metadata_filter
)
for chunk in response:
print(chunk)