如何一次处理多个JavaRDD？

Question

I had a large dataset of format csv and i need to perform some RDD operations on this dataset without using any DataFrames/Dataset API and SparkSQL.By achieving this, i loaded each column of data into separate JavaRDD.

Here is my Sample dataset:

id    name    address   rank
1001  john    NY        68
1002  kevin   NZ        72
1003  steve   WA        64

Here is the code i tried so far:

JavaRDD<String> diskfile = sc.textFile("/Users/hadoop/Downloads/a.csv");
JavaRDD<String> idRDD=diskfile.flatMap(line -> Arrays.asList(line.split(",")[0]));
JavaRDD<String> nameRDD=diskfile.flatMap(line -> Arrays.asList(line.split(",")[1]));
JavaRDD<String> addressRDD=diskfile.flatMap(line -> Arrays.asList(line.split(",")[2]));

after this i applied reduceByKey on both addressRDD and nameRDD like this:

JavaPairRDD<String,Integer> addresspair=address.mapToPair( t -> new Tuple2 <String,Integer>(t,1)).reduceByKey((x, y) -> x + y);
JavaPairRDD<String,Integer> namepair=nameRDD.mapToPair( t -> new Tuple2 <String,Integer>(t,1)).reduceByKey((x, y) -> x + y);

Problem:

I applied soryByVale(swap the key -values) on addresspair and get one address value( result ) which is occurred highest number of times. Now i need to return all required columns of csv file which contains address field as result .

Answer 1

You can use filter like below.

JavaRDD<String> filteredData = diskfile.filter(add -> add.contains(result));
filteredData.foreach(data -> {
            System.out.println(data);
        });