将嵌套json的spark列转换为字符串

Question

I'm relatively new to spark/scala and i've got the following problem i'm hoping you can help me out with. In order for my hashing algorithm to work i need to convert an array type field into a string. The schema below is similar to what i'm dealing with:

+-----------------+----------------+
| records         | Partition      |
+-----------------+----------------+
| [{data:[{...}..]| 20200101       |
+-----------------+----------------+
| [{data:[{...}..]| 20200102       |
+-----------------+----------------+

The field types are: {records: array, partition: string}

All i want is to convert the record field into a string, in the vein of:

[{data:[{...}..] --> "[{data:[{...}..]"

Any help on this would be greatly appreciated.

Thanks.

Answer 1

It would be useful to have your data schema (using df.printSchema ). But with simple example I managed to do it using a simple cast like this:

import org.apache.spark.sql.types._
val castDf = df.withColumn("ArrrayToString", $"myColName".cast(StringType))