来自不同数据格式的 Spark Dataframe

Question

I've this data set. for which I need to create a sparkdataframe in scala. This data is a column in a csv file. column name is dataheader

dataheader

"{""date_time"":""1999/05/22 03:03:07.011"",""cust_id"":""cust1"",""timestamp"":944248234000,""msgId"":""113"",""activityTimeWindowMilliseconds"":20000,""ec"":""event1"",""name"":""ABC"",""entityId"":""1001"",""et"":""StateChange"",""logType"":""type123,""lastActivityTS"":944248834000,""sc_id"":""abc1d1c9"",""activityDetectedInLastTimeWindow"":true}"
"{""date_time"":""1999/05/23 03:03:07.011"",""cust_id"":""cust1"",""timestamp"":944248234000,""msgId"":""114"",""activityTimeWindowMilliseconds"":20000,""ec"":""event2"",""name"":""ABC"",""entityId"":""1001"",""et"":""StateChange"",""logType"":""type123,""lastActivityTS"":944248834000,""sc_id"":""abc1d1c9"",""activityDetectedInLastTimeWindow"":true}"

I was able to read the csv file -

val df_tmp = spark
                    .read
                    .format("com.databricks.spark.csv")
                    .option("header","true")
                    .option("quoteMode", "ALL")
                    .option("delimiter", ",")
                    .option("escape", "\"")
                    //.option("inferSchema","true")
                    .option("multiline", "true")
                    .load("D:\\dataFile.csv")

I tried to split the data into separate columns in a dataframe but did not succeed.

one thing I noticed in data is both key and value are enclosed by double double quotes ""key1"":""value1""

Answer 1

If you want to get the field inside the data field, you need to parse it and write it into a new CSV file. It's obviously a string in json format