I’m getting a 403 permission denied error when trying to install the spark/pinecone connector on my databricks cluster, according to these docs: Databricks - Pinecone Docs.
I’m confused because the docs say to use an s3 path to install pinecone, but any s3 path I provide will point to the s3 account linked to my databricks workspace, presumably not pinecone’s s3?
Full error:
Library installation attempted on the driver node of cluster 1115-203734-kttv6dqs and failed. Please refer to the following error message or contact Databricks support. Error code: FAULT_OTHER, error message: java.util.concurrent.ExecutionException: java.nio.file.AccessDeniedException: s3a://pinecone-jars/1.1.0/spark-pinecone-uberjar.jar: getFileStatus on s3a://pinecone-jars/1.1.0/spark-pinecone-uberjar.jar: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://pinecone-jars.s3.us-east-1.amazonaws.com 1.1.0/spark-pinecone-uberjar.jar {} Hadoop 3.3.6, aws-sdk-java/1.12.390 Linux/5.15.0-1075-aws OpenJDK_64-Bit_Server_VM/25.412-b08 java/1.8.0_412 scala/2.12.15 kotlin/1.6.0 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 80FZQTEMSK8GYNJT, Extended Request ID: /DARMKvNQvlFHCNVGV+2hhgZS386tBMueWuhqaSBCti85/drVmumdMxV2vyfLnLZsnb105C9rVU=, Cloud Provider: AWS, Instance ID: i-01c99aa68497ca0d8 credentials-provider: com.amazonaws.auth.BasicSessionCredentials credential-header: AWS4-HMAC-SHA256 Credential=REDACTED_ACCESS_KEY(32a0c8de)/20250301/us-east-1/s3/aws4_request signature-present: true (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 80FZQTEMSK8GYNJT; S3 Extended Request ID: /DARMKvNQvlFHCNVGV+2hhgZS386tBMueWuhqaSBCti85/drVmumdMxV2vyfLnLZsnb105C9rVU=; Proxy: null), S3 Extended Request ID: /DARMKvNQvlFHCNVGV+2hhgZS386tBMueWuhqaSBCti85/drVmumdMxV2vyfLnLZsnb105C9rVU=:403 Forbidden