Spark（Java）：从文件名列表中获取文件名/内容对

Question

I'm currently working on a Spark project in Java, and I ran into a problem that I am not sure how to solve. I'm unfamiliar with the various join/union methods, so I figure one of those is the answer.

I currently want to input a list of filenames (or paths) and get a JavaPairRDD object, consisting of pairs of filenames/paths and text content.

I know I can use standard Java to get the text content and just input a List of filename-content tuples, but I feel that there has to be a "Spark" way of doing this.

I also know there is a wholeTextFile method but that only grabs everything in a directory and I'm not sure that will be the format I get (I might use Amazon S3 for example and I'm not sure if I can make the assumption about a directory there).

Furthermore, I am aware that I can parallelize each file separately in a loop, but how do I join these back together?

docs = //List<String> of document filenames
JavaRDD<String> documents = sc.parallelize(docs);
JavaPairRDD<String, String> = documents.???

Thanks in advance.

Edit : I am tempted to create a JavaPairRDD of <Filename, JavaRDD<String> filecontents> , but I'm not sure how to proceed from there. I'm also wary of this because it just sounds wrong (ie am I overriding the parallelism somehow?).

I know I could have Spark create a JavaRDD object from each document, convert them to List objects, and then feed them in as tuples, but is there a Spark specific way of doing this?

Edit 2 Apparently, I misunderstood how text files are loaded into a JavaRDD object. They don't load the entire string in as one object, they break it up by line. This makes me rethink my approach, as I do need things to break across lines for various reasons. So I'm thinking I have to go with the "hackish" approach of using spark to load the file and then converting it back into a List. However, I'll leave the question up in case somebody has a clever solution for this.

Answer 1

I'm going to switch to wholeTextFiles() instead, as I am running into more and more issues trying to get the data into the proper format.

Namely, I don't actually want the files to be broken up into lines, I want to break it up in a special way myself.

Answer 2

If you go by wholeTestFiles() way doesn't it would read whole data at once and then parallelize it over your standalone Spark cluster / workers ? Your driver code needs to be run at higher memory.

Answer 3

In Scala you can get the file name spark stream or spark sc using this query:

object GetFileNameFromStream extends java.io.Serializable {
   def getFileName(file: RDD[String]) :String ={
   file.toDebugString
  }
}