Spark Connector

Support
1

I have problem with loading data to pinecone with spark connector. I’ ve tried to follow this page from documentation. But in the article metadata is defined as plain string:

schema = StructType([  
    StructField("id",StringType(),True),  
    StructField("values",ArrayType(FloatType()),True),  
    StructField("namespace",StringType(),True),  
    StructField("metadata", StringType(), True),  
    StructField("sparse_values", StructType([  
        StructField("indices", ArrayType(IntegerType(), False), False),  
        StructField("values", ArrayType(FloatType(), False), False)  
    ]), True)  
])

In my case I want to define metadata as object with 2 fields, as:

StructField(
    "metadata",
    StructType(
        [
            StructField("A", ArrayType(StringType(), False), False),
            StructField("B", StringType(), False),
        ]
    ),
    True,
),

But after such a definition I got error when trying load data into Pinecone:
- Cannot write 'metadata': struct<A:array<string>,B:string> is incompatible with string