在RDD中嵌套使用sc.textFile

Question

I need to use Spark to process data in a large set of text files based on querying another index. I can do this for small cases (by converting the RDD to an array / see below), but am having difficulty in setting it up right to work with larger amounts of data.

I have this:

val rootPath = "..."
val ndxRDD = sc.textFile(rootPath + "index/2016-09-01*")

def ndxToDoc(articleName: String): String = { sc.textFile(rootPath + articleName).first(); }

// works
val artcilesArr = ndxRDD.collect().map(ndxToDoc);
val articlesRDD = sc.parallelize(articlesArr)

// does not work
// val articlesRDD = ndxRDD.map(ndxToDoc)

articlesRDD.count()

I believe the problem is that I am trying to read the file inside the rdd. How do I get the above working without an intermediate collect() - map -> textFile() - parallelize() set?

Thanks in advance!

Answer 1

I think this is the optimal approach for this kind of the task. Its your use case that wants it !

You must collect it as a list , otherwise you have to create a RDD inside a RDD which is not possible in the current implementation of Spark.

For more information why we cannot create RDD inside a RDD look here :

1. Spark-Google-Group Discussion

2. SPARK-NestedRDD

Hence this is an optimal approach, though I can suggest one thing is you can use OFFHeap memory to store large objects in memory