Convert datatype of cloumn from StringType to StructType in dataframe in spark scala

Question

|     ID|CO_ID|                           DATA|
+--------------------+--------------------+----+
|ABCD123|abc12|[{"month":"Jan","day":"monday"}] |
|BCHG345|wed34|[{"month":"Jul","day":"tuessay"}]|

I have dataframe above in which column DATA is of StringType.I want it to convert to StructType. How can I do this?

Answer 1

Use from_json

df.withColumn("data_struct",from_json($"data",StructType(Array(StructField("month", StringType),StructField("day", StringType)))))

On Spark 2.4.0, I get the following

import org.apache.spark.sql.types.{StructType, StructField, StringType}

val df = List ( ("[{\"month\":\"Jan\",\"day\":\"monday\"}]")).toDF("data")

val df2 = df.withColumn("data_struct",from_json($"data",StructType(Array(StructField("month", StringType),StructField("day", StringType)))))

df2.show

+--------------------+-------------+
|                data|  data_struct|
+--------------------+-------------+
|[{"month":"Jan","...|[Jan, monday]|
+--------------------+-------------+

df2.printSchema

root
 |-- data: string (nullable = true)
 |-- data_struct: struct (nullable = true)
 |    |-- month: string (nullable = true)
 |    |-- day: string (nullable = true)