我如何将平面数据帧转换为spark（scala或java）中的嵌套json

Question

I have sql query which returns the data set like this available in dataframe

id,type,name,ppu,batter.id,batter.type,topping.id,topping.type
101,donut,cake,0_55,1001,Regular,5001,None
101,donut,cake,0_55,1002,Chocolate,5001,None
101,donut,cake,0_55,1003,Blueberry,5001,None
101,donut,cake,0_55,1004,Devil's Food,5001,None
101,donut,cake,0_55,1001,Regular,5002,Glazed
101,donut,cake,0_55,1002,Chocolate,5002,Glazed
101,donut,cake,0_55,1003,Blueberry,5002,Glazed
101,donut,cake,0_55,1004,Devil's Food,5002,Glazed
101,donut,cake,0_55,1001,Regular,5003,Chocolate
101,donut,cake,0_55,1002,Chocolate,5003,Chocolate
101,donut,cake,0_55,1003,Blueberry,5003,Chocolate
101,donut,cake,0_55,1004,Devil's Food,5003,Chocolate

I need to cover this into a nested json structure like this.

{
    "id": "101",
    "type": "donut",
    "name": "Cake",
    "ppu": 0.55,
    "batter":
        [
            { "id": "1001", "type": "Regular" },
            { "id": "1002", "type": "Chocolate" },
            { "id": "1003", "type": "Blueberry" },
            { "id": "1004", "type": "Devil's Food" }
        ],
    "topping":
        [
            { "id": "5001", "type": "None" },
            { "id": "5002", "type": "Glazed" },
            { "id": "5003", "type": "Chocolate" }
        ]
}

do we have possibility to perform this in Dataframe aggregation or custom transformation i have to write.

Found similar question here Writing nested JSON in spark scala but doesnt have quite right answer.

Answer 1

So, apparently there is no straight way to do this task via the dataframe API. You could use the

df.toJson.{..}

but it wont give you the output you want.

You'll have to write a messy transform, I'd love to hear any other possible solutions. I'm assuming that your result fits in memory as it must be brought back to the driver. Also, I'm using Gson API to create the json here.

def arrToJson(arr: Array[Row]): JsonObject = {
    val jo = new JsonObject
    arr.map(row => ((row(0) + "," + row(1) + "," + row(2) + "," + row(3)),
      (row(4) + "," + row(5) + "," + row(6) + "," + row(7))))
      .groupBy(_._1).map(f => (f._1.split(","), f._2.map(_._2.split(","))))
      .foreach { x =>
        {

          jo.addProperty("id", x._1(0))
          jo.addProperty("type", x._1(1))
          jo.addProperty("name", x._1(2))
          jo.addProperty("ppu", x._1(3))

          val bja = new JsonArray
          val tja = new JsonArray
          x._2.foreach(f => {
            val bjo = new JsonObject
            val tjo = new JsonObject

            bjo.addProperty("id", f(0))
            bjo.addProperty("type", f(1))

            tjo.addProperty("id", f(2))
            tjo.addProperty("type", f(3))

            bja.add(bjo)
            tja.add(tjo)
          })
          jo.add("batter", bja)
          jo.add("topping", tja)

        }
      }

    jo
  }