Spark-groupByKey其他选项

Question

We have a need to control huge dataset using Spark. The control consists of grouping the data by key (we use for this : groupByKey() ) and then looping for each grouped data to check coherence between them.

For example we have the below csv file contains these columns to check :

id;dateBegin;dateEnd;event;dateEvent
1;12/02/2015;30/05/2015;active;05/04/2015
1;12/06/2015;30/07/2015;dead;05/07/2015
2;12/02/2016;30/07/2016;dead;05/04/2015

We used JavaRdd<String>.map().groupByKey() but Spark freezes for large dataset.

Are there other options to use? Thank you

Answer 1

According to this documentation: Avoid GroupByKey

reduceByKey() is good for larger data sets as, Spark (before shuffling data) can combine the output with a common key. But, groupByKey() shuffles the data (kv pairs) creating unnecessary data sets.

Look for better (other) alternatives to groupByKey . Like,

• combineByKey
• foldByKey