Hadoop Spark：如何在JavaRDD中创建不同的元素？

Question

I want to store distinct of some JavaRDD collection to a file in Spark?

By using distinct() method of RDD, I couldn't achieve the same.

My guess is RDD treats each element as a individual instance. How can we achieve the distinct in this case.

Following is code snippet, Can anyone please help?

public static void main(String[] args) {
    SparkConf conf = new SparkConf().setAppName("Xml Spark Demo");
    JavaSparkContext sc = new JavaSparkContext(conf);

    JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(
            sc);


    // Load a text file and convert each line to a JavaBean.
    JavaRDD<String> dataFromFile = sc.textFile(
            "/home/kedarnath/Rentals/inputData/temp-01.xml").map(
            new ParseAgentFromXml());

    //Need distinct values here
    dataFromFile.distinct().saveAsTextFile("/home/kedarnath/Rentals/testOutputDistinct.txt");

}

Thanks in advance,

~Kedar

Answer 1

I am not sure if this would be the most efficient way of doing it from a performance perspective, but I would try dividing the process in two different steps: dinstinct and mapping to pair. Consider the following example:

Original dataset:          Desired output (distinct elements)

Apple                      1, Apple
Tree                       2, Tree
Car                        3, Car
Priest                     4, Priest
Apple                      5, Phone
Tree
Apple
Phone

• Distinct:

By using the distinct() transformation, you would obtain a new RDD dataset with all the distinct element. In this case, it would return something like:

Apple
Tree
Car
Priest
Phone

• Map to pair:

The next step would be to associate a key to every value in the RDD dataset, transforming it into a key-value format. For this, the transformation mapToPair() could be used. The output would result on the desired output.

1, Apple
2, Tree
3, Car
4, Priest
5, Phone

Visit this page to get more information about the different available methods.