read json into multiple spark dataframes using scala

Question

my json structure is something like this:

{
  "posts": [],
  "persons": [],
  "organizations": [],
  "meta": {
    "sources": [
      "http://loksabha.nic.in/",
      "http://wikidata.org/",
      "http://gender-balance.org/"
    ]
  },
  "memberships": [],
  "events": [],
  "areas": []
}

i want to read posts into a dataframe, wehre posts is an array of json objects. similarly other json arrays, except "meta". "sources" array inside "meta" json object should be read into another dataframe.

Is there anyway to achieve this with spark scala.

Any help is greatly appreciated.

Thanks in advance Shakti

Answer 1

You could use the expand function. I guess you have something like

val jsonDf = spark.read.json("your_json.json")
val postsDf = jsonDF.withColumn("post", explode(col("posts")).select("post")