Parse & flatten JSON object in a text file using Spark & Scala into Dataframe

Question

I have a text file with below structure.

(employeeID: Int, Name: String, ProjectDetails: JsonObject{[{ProjectName, Description, Duriation, Role}]})

Eg:

(123456, Employee1, {“ProjectDetails”:[ 
                                       { “ProjectName”: “Web Develoement”, “Description” : “Online Sales website”, “Duration” : “6 Months” , “Role” : “Developer”}
                                       { “ProjectName”: “Spark Develoement”, “Description” : “Online Sales Analysis”, “Duration” : “6 Months” , “Role” : “Data Engineer”}
                                       { “ProjectName”: “Scala Training”, “Description” : “Training”, “Duration” : “1 Month” }
                                       ]
                     }

Could someone help me to parse & flatten the record as below dataframe using scala?

employeeID, Name, ProjectName, Description, Duration, Role
123456, Employee1, Web Develoement, Online Sales website, 6 Months , Developer
123456, Employee1, Spark Develoement, Online Sales Analysis, 6 Months, Data Engineer
123456, Employee1, Scala Training, Training, 1 Month, null

Answer 1

You can try this .. But slightly modified the input structure since first two columns were not in Json format.

 scala> import org.apache.spark.SparkConf scala> import org.apache.spark.SparkContext scala> import org.apache.spark.sql.SQLContext scala> import org.apache.spark.sql._ scala> val sqlSC = new org.apache.spark.sql.SQLContext(sc) scala> import sqlSC.implicits._ scala> val emp_DF =sqlSC.jsonFile("file:///C:/Users/ABCD/Desktop/Examples/Spark/Mailing List/Employee_Nested_Projects.json") scala> case class ProjectInfo(ProjectName:String,Description:String,Duration:String,Role:String) scala> case class Project(employeeID:Int,Name:String,ProjectDetails:Seq[ProjectInfo]) scala> val emp_projects_DF = emp_DF.explode(emp_DF("ProjectDetails")) { case Row(x: Seq[Row])=> x.map(x=> ProjectInfo(x(0).asInstanceOf[String],x(1).asInstanceOf[String],x(2).asInstanceOf[String],x(3).asInstanceOf[String]))} scala> emp_projects_DF.select($"employeeID",$"Name",$"ProjectName",$"Description",$"Duration",$"Role").show()