convert a column with json value to a data frame using scala spark

Question

I found several helpful answers but that were all converting son file to df, in my case, I have a df with columns with son in them, like this:

s-timestamp : 2019-10-10

content : {"META":{"testA":"1","TABLENAME":"some_table_name"},"PINACOLADA":{"sampleID":"0","itemInserted":"2019-10-10","sampleType":"BASE",}"

I need to normalize the content column, how can I do that.

Answer 1

Welcome. There are a few ways of dealing with JSON strings in Spark DF columns. You can use functions like get_json_object to extract specific fields from your JSON or from_json to transform the field into a StructType with a given schema. Another option is to use spark.read.json to parse and create a separate dataframe from the column's contents. Have a look at my solution here and let me know if it helps.