关于Tuple2的RDD的Apache Spark forEach：为我的RDD中的所有Tuple2返回一个值

Question

I have this code that's been giving me an unexpectedly wrong result that i couldn't solve :

// A method that calls the collectDataRDD(logValues, rowData) method :

// ....
// my collectDataRDD(Values, rowData) method : 

The problem is that when i try to run methods like getStatus() or getValidationDate() on Data Objects which are the values of my Tuple2, it only gives one output for all the objects in my JavaRDD which is wrong, because the JavaRDD contains multiple different Objects. However when i checked the keys of my tuple2 it gave me correct results.

I have tried everything and still couldn't figure it out. Can anyone please help me solve this. THanks a lot in advance.

Answer 1

Verify if

ticketsrdd.foreach((Tuple2<String, Data> rowData) -> {
    collectLogDataRDD(logValues, rowData);
}

is what you want to do. This function is called for each element one by one and Tuple2 will have only one entry in that case.

Answer 2

JavaRDD<Tuple2<String, Data>> ticketsrdd=TransformToRDD.transformToRDD(transformer.transform());
DataStore.setData(tickets);

Will be kind of Map<String, Tuple2> . And your Tuple2 will have one key as String and one value as Data.

Now when you say Data ticket = rowData._2 ; you are getting 1 Data object from 1 tuple. So for each tuple in ticketsrdd its going to called collectLogDataRDD .

Lets say ticketsrdd has 100 element then its going to call collectLogDataRDD 100 times, and each time ticket.getStatus(); will also be be call.

This is what code is doing. What different behavior do you expect?