Apache Spark 最便宜的触发 RDD 转换的方法

Question

I'm totally new to Apache Spark and I've set up a standalone cluster to run a sorting algorithm for big amounts of data (Integer numbers).

I have it working the way I want. The core is as follows:

JavaRDD<Integer> rdd = ctx
                .parallelize(Collections.<Integer>emptyList(), PARTITIONS)
                .mapPartitions(partition ->
                        ThreadLocalRandom
                                .current()
                                .ints(NUMBERS_PER_PARTITION, Integer.MIN_VALUE, Integer.MAX_VALUE)
                                .boxed()

                                .parallel()
                                .collect(Collectors.toList()))
                .sortBy(x -> x, true, PARTITIONS);

This will generate random numbers in the cluster and then sort them.

The problem is that I am only interested in the sorting time for an experiment, but Spark is lazy and the sorting will only be triggered with a given action. I'm using count() to trigger the sorting, but it takes a very long time to finish the counting, therefore it delays my experiment. I don't care about getting the sorted numbers, or even a sample of it, since I already know it's sorting correctly.

Is there a way that I can trigger the .sortBy() without having to wait for the action that triggered it to finish? And if there isn't, is there a cheaper action than count() ?

Answer 1

sort is a lazy spark transformations
you can use one of the non lazy return values to trigger the action
you already tried count which is taking a lot of time
try: first() or take(n)

Here is a list of lazy / non-lazy actions

https://www.mapr.com/ebooks/spark/apache-spark-cheat-sheet.html