We have a python script which loads files locally stored PDF’s into a pinecone assistant - nothing too clever .
After what appears to be a random number of API calls and PDFs uploaded successfully, it starts to throw this error. If I stop it and process the file individually, it’s fine. If I restart it, it’s obviously fine, but it’s just weird. I’m trying to load about 1000 PDFs and after about 50 or so, it starts to then throw this error, so it’s like some weird limit
def upload_to_pinecone(assistant, file_path: Path, metadata: dict) -> bool:
"""Upload a file to Pinecone Assistant with metadata."""
try:
response = assistant.upload_file(
file_path=str(file_path), metadata=metadata, timeout=None
)
print(f"Uploaded successfully {response}")
return True
except Exception as e:
print(f" Upload failed: {e}")
return False