Grpc StatusRuntimeException: UNAVAILABLE: io exception using Spark

If I try to upsert to the pinecone index (Pod) using the following method (referred from Databricks - Pinecone Docs)

(
            df.write
            .option("pinecone.apiKey", api_key)
            .option("pinecone.indexName", index_name)
            .option("pinecone.projectName", project_name)
            .option("pinecone.environment", environment)
            .format("io.pinecone.spark.pinecone.Pinecone")
            .mode("append")
            .save()
        )

I am getting

org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 4584.0 failed 4 times, most recent failure: Lost task 5.3 in stage 4584.0 (TID 1437233) (executor 683): io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
File <command-2093907735488114>, line 24
     14 for attempt in range(max_retries):
     15     try:
     16         (
     17             df.write
     18             .option("pinecone.apiKey", api_key)
     19             .option("pinecone.indexName", index_name)
     20             .option("pinecone.projectName", project_name)
     21             .option("pinecone.environment", environment)
     22             .format("io.pinecone.spark.pinecone.Pinecone")
     23             .mode("append")
---> 24             .save()
     25         )
     26         break  # Exit the loop if the write operation is successful
     27     except Exception as e:

File /databricks/spark/python/pyspark/instrumentation_utils.py:47, in _wrap_function.<locals>.wrapper(*args, **kwargs)
     45 start = time.perf_counter()
     46 try:
---> 47     res = func(*args, **kwargs)
     48     logger.log_success(
     49         module_name, class_name, function_name, time.perf_counter() - start, signature
     50     )
     51     return res

File /databricks/spark/python/pyspark/sql/readwriter.py:1679, in DataFrameWriter.save(self, path, format, mode, partitionBy, **options)
   1677     self.format(format)
   1678 if path is None:
-> 1679     self._jwrite.save()
   1680 else:
   1681     self._jwrite.save(path)

File /databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:1355, in JavaMember.__call__(self, *args)
   1349 command = proto.CALL_COMMAND_NAME +\
   1350     self.command_header +\
   1351     args_command +\
   1352     proto.END_COMMAND_PART
   1354 answer = self.gateway_client.send_command(command)
-> 1355 return_value = get_return_value(
   1356     answer, self.gateway_client, self.target_id, self.name)
   1358 for temp_arg in temp_args:
   1359     if hasattr(temp_arg, "_detach"):

File /databricks/spark/python/pyspark/errors/exceptions/captured.py:188, in capture_sql_exception.<locals>.deco(*a, **kw)
    186 def deco(*a: Any, **kw: Any) -> Any:
    187     try:
--> 188         return f(*a, **kw)
    189     except Py4JJavaError as e:
    190         converted = convert_exception(e.java_exception)

File /databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
    324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325 if answer[1] == REFERENCE_TYPE:
--> 326     raise Py4JJavaError(
    327         "An error occurred while calling {0}{1}{2}.\n".
    328         format(target_id, ".", name), value)
    329 else:
    330     raise Py4JError(
    331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
    332         format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling o478.save.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 4584.0 failed 4 times, most recent failure: Lost task 5.3 in stage 4584.0 (TID 1437233) ([host IP] executor 683): io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
	at

Hi @shyamSiyer and welcome to the Pinecone community forums!

Thank you for your question.

Could you please share all your relevant code, being careful not to include any secrets such as your Pinecone API key?

In the meantime, I’m reaching out to a team member who worked on the databricks integration.

Best,
Zack

Hi @shyamSiyer,

I heard back from one of our engineers closest to the connector that passing projectname and env is not valid anymore.

Could you please share:

  1. The version of the Spark connector you’re using
  2. The link to the documentation that you’re following currently?

Best,
Zack

But that passing without projectname and env is erroring out. They are asking for project name and env.
Databricks Runtime: 14.2.x-gpu-ml-scala2.12
Spark Version: 3.5.0
The link I am following is already mentioned (Databricks - Pinecone Docs), the only difference is that I am using model all-mpnet-base-v2.

Hi @shyamSiyer,

I lead the spark connector work and I’m curious to know what spark connector version you’re using i.e. which pinecone’s spark connector assembly jar you imported via s3 bucket in the databricks environment?

Also I dont see projectName and environment on the Databricks-Pinecone Docs, so would you be kind enough to provide the exact link to that doc? The doc mentions the following:

embeddings_df.write  
    .option("pinecone.apiKey", api_key) 
    .option("pinecone.indexName", index_name)  
    .format("io.pinecone.spark.pinecone.Pinecone")  
    .mode("append")  
    .save()  

Lastly, would you please provide the whole codebase and not just the df.write command.

Thanks :slight_smile:

Hi Rohan
I am using the spark-pinecone-uberjar from dbfs (dbfs:/FileStore/lib/spark-pinecone-uberjar.jar). I am using a databricks notebook and these installations have already been done on the provided compute machine, just for your information. But when I try to install a new jar file on my machine using the s3 path shown on the site (databricks compute machine), I am getting a permission error. Here is the code (Google Colab)
Details on jar file
Manifest-Version: 1.0
Implementation-Title: spark-test
Implementation-Version: 0.1.0-SNAPSHOT
Specification-Vendor: spark-test
Specification-Title: spark-test
Implementation-Vendor-Id: spark-test
Specification-Version: 0.1.0-SNAPSHOT
Main-Class: org.example.pinecone.MainJob
Implementation-Vendor: spark-test

Hi @shyamSiyer,
I’m assuming your compute was setup by manually uploading the assembly/uber jar to the dbfs in dbr environment? If so, you can upload a new one since projectName and environment were accepted before v1.0.0 release, so please use the latest assembly jar.

If you’re trying to use the s3 bucket to import the jar, please follow the instructions on the README.

Also note that the dbr environment does require importing the assembly jar because of the way dependency management works in dbr.

Thanks!

Thank you for your reply. I tried to install s3://pinecone-jars/1.1.0/spark-pinecone-uberjar.jar. This is the error I am facing.

DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: java.util.concurrent.ExecutionException: java.nio.file.AccessDeniedException: s3a://pinecone-jars/1.1.0/spark-pinecone-uberjar.jar: getFileStatus on s3a://pinecone-jars/1.1.0/spark-pinecone-uberjar.jar: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://pinecone-jars.s3.us-east-1.amazonaws.com 1.1.0/spark-pinecone-uberjar.jar {} Hadoop 3.3.6, aws-sdk-java/1.12.610 Linux/5.15.0-1063-aws OpenJDK_64-Bit_Server_VM/25.392-b08 java/1.8.0_392 scala/2.12.15 kotlin/1.6.0 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy

Hi,

I have attached the screenshot for installing the uber jar via s3 and I am successfully able to upsert data without any read access error. So would you please let me know how you tried installing the library and if installing it in the way described below is still giving you issues?

Thanks!