Spark RDD，如何生成长度为N的JavaRDD？

Question

(part of problem is docs that say "undocumented" on parallelize leave me reading books for examples that don't always pertain )

I am trying to create an RDD length N = 10^6 by executing N operations of a Java class we have, I can have that class implement Serializable or any Function if necessary. I don't have a fixed length dataset up front, I am trying to create one. Trying to figure out whether to create a dummy array of length N to parallelize, or pass it a function that runs N times.

Not sure which approach is valid/better, I see in Spark if I am starting out with a well defined data set like words in a doc, the length/count of those words is already defined and I just parallelize some map or filter to do some operation on that data.

In my case I think it's different, trying to parallelize the creation an RDD that will contain 10^6 elements...

DESCRIPTION:

In Java 8 using Spark 1.5.1, we have a Java method doDrop() that takes a PipeLinkageData and returns a DropResult.

I am thinking I could use map() or flatMap() to call a one to many function, I was trying to do something like this in another question that never quite worked :

JavaRDD<DropResult> simCountRDD = spark.parallelize(makeRange(1,getSimCount())).map(new Function<Integer, DropResult>()
    {
      public DropResult call(Integer i) { 
         return pld.doDrop(); 
      }
    });

Thinking something like this is more the correct approach?

    // pld is of type PipeLinkageData, it's already initialized

    // parallelize wants a collection passed into first param
    List<PipeLinkageData> pldListofOne = new ArrayList();

    // make an ArrayList of one
    pldListofOne.add(pld);

    int howMany = 1000000;

    JavaRDD<DropResult> nSizedRDD = spark.parallelize(pldListofOne).flatMap(new FlatMapFunction<PipeLinkageData, DropResult>() 
        { 
            public Iterable<DropResult> call(PipeLinkageData pld) {

                List<DropResult> returnList = new ArrayList();

                // is Spark good at spreading a for loop like this?
                for ( int i = 0; i < howMany ; i++ ){
                    returnList.add(pld.doDrop());  
                }

                // EDIT changed from returnRDD to returnList
                return returnList;
            }

            }); 

One other concern: A JavaRDD is corrrect here? I can see needing to call FlatMapFunction but I don't need a FlatMappedRDD? And since I am never trying to flatten a group of arrays or lists to a single array or list, do I really ever need to flatten anything?

Answer 1

1. The first approach should work as long as DropResult and can be serialized PipeLinkageData and there are no issues with its internal logic (like depending on a shared state).

2. The second approach in a current form doesn't make sense. A single record will be processed on a single partition. It means a whole process will be completely sequential and can crash if data doesn't fit in a single worker memory. Increasing number of elements should solve the problem but it doesn't improve on the first approach

3. Finally you can initialize an empty RDD and then use mapPartititions replacing FlatMapFunction with almost identical MapPartitionsFunction and generate required number of objects per partition.