"Connection reset by peer" during indexation of vectors

Support
1

Hi,

I ran into this error (multiple times) when doing the indexation of vectors, it occurs randomly on some vectors and when I retry indexing them the error disappears. Does anyone have an explanation please ?

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/techdealpartners/ai/embeddings/modules/insert.py", line 94, in insert_chunk_data
    self.pinecone_index.upsert(vectors)
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/utils/error_handling.py", line 17, in inner_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pinecone/index.py", line 59, in upsert
    return self._vector_api.upsert(
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/client/api_client.py", line 776, in __call__
    return self.callable(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/client/api/vector_operations_api.py", line 593, in __upsert
    return self.call_with_http_info(**kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/client/api_client.py", line 838, in call_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/client/api_client.py", line 413, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/client/api_client.py", line 200, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/client/api_client.py", line 459, in request
    return self.rest_client.POST(url,
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/client/rest.py", line 271, in POST
    return self.request("POST", url,
  File "/usr/local/lib/python3.9/dist-packages/pinecone/core/client/rest.py", line 157, in request
    r = self.pool_manager.request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Thank you !

2

Hey @Chaymae, I think this is just an error with the client having trouble establishing a conneciton that may happen intermittently depending on your network. It’s not related to any vectors.
I would suggest trying to use the gRPC client to see if that improves things for you. Let me know how it goes!

3

Hi, I am having the same issue.
I used the code in https://www.pinecone.io/docs/examples/semantic-text-search/ and it did work initially but after deleting and recreating the index it ceased to work permanently and I keep getting this error.
I also tried installing the grpc library but it didn’t resolve this and it looks like the call is made using the rest client anyway.

Traceback (most recent call last):
  File "/Users/anonymous/Code/pinecone-poc/test_pinecone2.py", line 65, in <module>
    index.upsert(vectors=zip(batch_df.vector_id, batch_df.title_vector), namespace='title')
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/utils/error_handling.py", line 17, in inner_func
    return func(*args, **kwargs)
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/index.py", line 70, in upsert
    return self._vector_api.upsert(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/api_client.py", line 776, in __call__
    return self.callable(self, *args, **kwargs)
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/api/vector_operations_api.py", line 835, in __upsert
    return self.call_with_http_info(**kwargs)
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/api_client.py", line 838, in call_with_http_info
    return self.api_client.call_api(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/api_client.py", line 413, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/api_client.py", line 200, in __call_api
    response_data = self.request(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/api_client.py", line 459, in request
    return self.rest_client.POST(url,
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/rest.py", line 271, in POST
    return self.request("POST", url,
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/rest.py", line 157, in request
    r = self.pool_manager.request(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/anonymous/.pyenv/versions/3.8.7/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/anonymous/.pyenv/versions/3.8.7/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/Users/anonymous/.pyenv/versions/3.8.7/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

stack shows call to rest client in line:

File "/Users/anonymous/Code/pinecone-poc/.venv/lib/python3.8/site-packages/pinecone/core/client/api_client.py", line 459, in request
4

Hi Itaim,
Looking through the example code, it looks like the batch size for upserting is set to 300 which could be the cause. Try lowering the batch size to say 100 and see if that intermittent error goes away. Also, just to be sure, also update your client if you haven’t already by using ‘pip install -U “pinecone-client[grpc]”’

5

Hi Kbutler,

I tried lowering the batch size to 100 and then 10, also installed pinecone-client[grpc], current version is 2.0.11.
But was not successful.

6

Please review the following suggestions for performance related issues and let us know if it helps.

Basic performance checklist

  • Switch to a cloud environment. For example: EC2, GCE, Google Colab, GCP AI Platform Notebook, or SageMaker Notebook. If you experience slow uploads or high query latencies, it might be because you are accessing Pinecone from your home network.
  • Deploy your application and your Pinecone service in the same region. For users on the Free plan, Pinecone runs in GCP US-West (Oregon). Contact us if you need a dedicated deployment.
  • Reuse connections. We recommend you reuse the same pinecone.Index() instance when you are upserting and querying the same index.
  • Operate within known limits.
7

Thanks! Works on SageMaker

8

FYI. We have released a new pinecone-client and ‘pinecone-client[grpc]’. It includes better connection timeout handling. Please upgrade.

9

I have the same problem, anybody sovled that ?