Hello Pinecone Community, I’m experiencing a persistent issue with SSL/TLS connections when attempting to access Pinecone from my Flask application hosted on Google Cloud Run – the connection is unexpectedly closed, resulting in an SSLZeroReturnError.
Error Message: SSLError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129)
My application and Pinecone are hosted in GCP’s Iowa (us-central1) region.
Here’s a summary of my setup and what I’ve tried so far:
- Application Stack: Flask (python), hosted on GCP Cloud Run, using Gunicorn as the WSGI server
- Pinecone Version: pinecone-client==2.2.4
- SSL/TLS Verification: Originally enabled (default), but disabled for testing purposes with no resolution to the issue
- Flask Development Server: I considered reverting to Flask’s development server for testing, but I stuck with Gunicorn for its production readiness.
- Gunicorn Timeout: Increased to 3600 seconds
- Requests to Pinecone: Made within a ThreadPoolExecutor in Flask to avoid blocking the main thread, but the SSL/TLS connection still closes unexpectedly. I’ve reviewed my GCP networking and firewall settings, and there are no obvious VPC service controls that could be interfering with the connection.
- Moved my application Docker image from python 3.9:slim to python 3.9, to 3.10 but no chance.
- Added certs download in Dockerfile
apt-get install -y --no-install-recommends ca-certificates && \ - Performing curl calls to my app locally with and without the Docker image works as expected
- I tried varying the Cloud Run instance’s memory from 8 GB to 32 GB but to no avail.
- I just rotated my Pinecone API key, works locally, still doesn’t work on Cloud Run
- I am still on the free version, using gcp-starter. To the best of my knowledge, this is not a limitation of the free version.
- I downgraded from
urllib3==1.26.18tourllib3==1.26.12(seen on a github issue) - Set number of workers in multi-threading to 1
- Disabled SSL checks with
os.environ['PYTHONHTTPSVERIFY'] = '0'
Here’s the traceback:
Traceback (most recent call last):
File "/app/src/Llama_index_sandbox/custom_react_agent/tools/query_engine.py", line 13, in call
response = self._query_engine.query(query_str)
File "/usr/local/lib/python3.9/site-packages/llama_index/indices/query/base.py", line 23, in query
return self._query(str_or_query_bundle)
File "/usr/local/lib/python3.9/site-packages/llama_index/query_engine/retriever_query_engine.py", line 171, in _query
nodes = self.retrieve(query_bundle)
File "/usr/local/lib/python3.9/site-packages/llama_index/query_engine/retriever_query_engine.py", line 123, in retrieve
nodes = self._retriever.retrieve(query_bundle)
File "/usr/local/lib/python3.9/site-packages/llama_index/indices/base_retriever.py", line 22, in retrieve
return self._retrieve(str_or_query_bundle)
File "/usr/local/lib/python3.9/site-packages/llama_index/indices/vector_store/retrievers/retriever.py", line 87, in _retrieve
return self._get_nodes_with_embeddings(query_bundle)
File "/usr/local/lib/python3.9/site-packages/llama_index/indices/vector_store/retrievers/retriever.py", line 164, in _get_nodes_with_embeddings
query_result = self._vector_store.query(query, **self._kwargs)
File "/usr/local/lib/python3.9/site-packages/llama_index/vector_stores/pinecone.py", line 327, in query
response = self._pinecone_index.query(
File "/usr/local/lib/python3.9/site-packages/pinecone/core/utils/error_handling.py", line 17, in inner_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/pinecone/index.py", line 455, in query
response = self._vector_api.query(
File "/usr/local/lib/python3.9/site-packages/pinecone/core/client/api_client.py", line 776, in __call__
return self.callable(self, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/pinecone/core/client/api/vector_operations_api.py", line 716, in __query
return self.call_with_http_info(**kwargs)
File "/usr/local/lib/python3.9/site-packages/pinecone/core/client/api_client.py", line 838, in call_with_http_info
return self.api_client.call_api(
File "/usr/local/lib/python3.9/site-packages/pinecone/core/client/api_client.py", line 413, in call_api
return self.__call_api(resource_path, method,
File "/usr/local/lib/python3.9/site-packages/pinecone/core/client/api_client.py", line 200, in __call_api
response_data = self.request(
File "/usr/local/lib/python3.9/site-packages/pinecone/core/client/api_client.py", line 459, in request
return self.rest_client.POST(url,
File "/usr/local/lib/python3.9/site-packages/pinecone/core/client/rest.py", line 271, in POST
return self.request("POST", url,
File "/usr/local/lib/python3.9/site-packages/pinecone/core/client/rest.py", line 157, in request
r = self.pool_manager.request(
File "/usr/local/lib/python3.9/site-packages/urllib3/request.py", line 78, in request
return self.request_encode_body(
File "/usr/local/lib/python3.9/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/usr/local/lib/python3.9/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='quickstart-377ec93.svc.gcp-starter.pinecone.io', port=443): Max retries exceeded with url: /query (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)')))
Some sources: