Loading json data into Pair RDD in Spark using java

Question

I am very new to Spark.

I have a very basic question. I read a file in Spark RDD in which each line is a JSON. I want to make apply groupBy like transformations. So I want to transform each JSON line into a PairRDD . Is there a straight forward way to do it in Java ?

My json is like this:

{
        "tmpl": "p",
        "bw": "874",
        "aver": {"cnac": "US","t1": "2"},
}

Currently, the way I am trying is the to split by , first and then by : . Is there any straight forward way to do this?

My current code:

val pairs = setECrecords.flatMap(x => (x.split(",")))
pairs.foreach(println)

val pairsastuple = pairs.map(x => if(x.split("=").length>1) (x.split("=")(0), x.split("=")(1)) else (x.split("=")(0), x))

Answer 1

You can try mapToPair() , but using the Spark SQL & DataFrames API will enable you to group things much more easily. The data frames API allows you to load JSON data directly.