如何在SparkContext.wholeTextFiles之后分别处理多个文件？

Question

I'm trying to use wholeTextFiles to read all the files names in a folder and process them one-by-one seperately(For example, I'm trying to get the SVD vector of each data set and there are 100 sets in total). The data are saved in .txt files spitted by space and arranged in different lines(like a matrix).

The problem I came across with is that after I use "wholeTextFiles("path with all the text files")", It's really difficult to read and parse the data and I just can't use the method like what I used when reading only one file. The method works fine when I just read one file and it gives me the correct output. Could someone please let me know how to fix it here? Thanks!

public static void main (String[] args) {
    SparkConf sparkConf = new SparkConf().setAppName("whole text files").setMaster("local[2]").set("spark.executor.memory","1g");;
    JavaSparkContext jsc = new JavaSparkContext(sparkConf);
    JavaPairRDD<String, String> fileNameContentsRDD = jsc.wholeTextFiles("/Users/peng/FMRITest/regionOutput/");

    JavaRDD<String[]> lineCounts = fileNameContentsRDD.map(new Function<Tuple2<String, String>, String[]>() {
        @Override
        public String[] call(Tuple2<String, String> fileNameContent) throws Exception {
                String content = fileNameContent._2();
                String[] sarray = content .split(" ");
                double[] values = new double[sarray.length];
                for (int i = 0; i< sarray.length; i++){
                    values[i] = Double.parseDouble(sarray[i]);
                }


            pd.cache();
            RowMatrix mat = new RowMatrix(pd.rdd());

            SingularValueDecomposition<RowMatrix, Matrix> svd = mat.computeSVD(84, true, 1.0E-9d);
            Vector s = svd.s();
    }});

Answer 1

Quoting the scaladoc of SparkContext.wholeTextFiles :

wholeTextFiles(path: String, minPartitions: Int = defaultMinPartitions): RDD[(String, String)] Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI. Each file is read as a single record and returned in a key-value pair, where the key is the path of each file, the value is the content of each file.

In other words, wholeTextFiles might not simply be what you want.

Since by design "Small files are preferred" (see the scaladoc), you could mapPartitions or collect (with filter ) to grab a subset of the files to apply the parsing to.

Once you have the files per partitions in your hands, you could use Scala's Parallel Collection API and schedule Spark jobs to execute in parallel :

Inside a given Spark application (SparkContext instance), multiple parallel jobs can run simultaneously if they were submitted from separate threads. By “job”, in this section, we mean a Spark action (eg save, collect) and any tasks that need to run to evaluate that action. Spark's scheduler is fully thread-safe and supports this use case to enable applications that serve multiple requests (eg queries for multiple users).

By default, Spark's scheduler runs jobs in FIFO fashion. Each job is divided into “stages” (eg map and reduce phases), and the first job gets priority on all available resources while its stages have tasks to launch, then the second job gets priority, etc. If the jobs at the head of the queue don't need to use the whole cluster, later jobs can start to run right away, but if the jobs at the head of the queue are large, then later jobs may be delayed significantly.