How to apply flatmap on JSONArray and convert to JSONObject in Spark?

Question

I am working on spark java module, where I make a geocode api (batch request of 100 records). Now for every request I will get JSONArray of JSONObject in one row like as below.

Usually there will be 100 Address, but i gave only 2 addresses

  Locations        | Batch_Address
------------------------------------------
Address1,Address2  | [ { name:Address1,lat:12.89,lng:23.56} , { name:Address2,lat:12.3,lng:23.4} ]

What I required is like result as below

Address
----------
{ name:Address1,lat:12.89,lng:23.56}
{ name:Address2,lat:12.3,lng:23.4}

Answer 1

If your code deals with java8, which has java.util.Spliterator<T> , so you can use it to create list from jsonarray.

String type

JavaRDD<String> JsonObject = json_Batch.javaRDD().flatMap(f -> {
            String res = f.getAs("Batch_Address").toString();


             JSONArray jsonArr = new JSONArray(res);


            List<String> list = StreamSupport.stream(jsonArr.spliterator(), false)
                    .map(val ->   val.toString())

                    .collect(Collectors.toCollection( ArrayList::new ));

            return list.iterator();

        }); 

        JsonObject.foreach(f -> System.out.println(f));

JSONObject type

  JavaRDD<JSONObject> JsonObject = json_Batch.javaRDD().flatMap(f -> {
            String res = f.getAs("Batch_Address").toString();


             JSONArray jsonArr = new JSONArray(res);


            List<JSONObject> list = StreamSupport.stream(jsonArr.spliterator(), false)
                    .map(val ->  (JSONObject) val)

                    .collect(Collectors.toCollection( ArrayList::new ));

            return list.iterator();

        }); 

        JsonObject.foreach(f -> System.out.println(f));