如何使用 Spark 读取文件夹文件？

Question

I have a hdfs folder, in this folder has many files txt. I want to read content in these files using spark.

My code:

// Create spark session
    val spark = SparkSession.builder()
                  .master("spark://master:7077")
                  .appName("Indexing data to elasticsearch")
                  .getOrCreate()

    spark.sparkContext.setLogLevel("ERROR")

    // Read folder file
    val df:DataFrame = spark.read.text("hdfs://master:9000/user/file/shakespeare")

I want to get the content of each file in my folder from DataFrame. How should I do?

Answer 1

This is one of the many cases where RDDs seem to be way more flexible than DataFrames, because RDDs provide the wholeTextFiles method.

The wholeTextFiles method is basically like textFile but instead of reading the input per lines in all of the files, it reads and stores a record/PairRDD/key-value pair per file . The schema of the result RDD will be like this:

(path_to_file, file_contents)

(However, one must be quite careful using this method because you probably can't predict the length of the contents of the files under a directory, so this can result into running out of memory).

So let's say we have a number of text files ( a.txt , b.txt , ...), each named by the alphabet letter that they have in their contents like shown below:

Using the wholeTextFiles method will result in the following RDD pairs (where you might wanna get rid of the full path of each file as the key of each pair for better readability):

(hdfs:/.../.../a.txt,a aa aaa aaaa aaaaa)
(hdfs:/.../.../b.txt,b bb bbb bbbb bbbbb)
(hdfs:/.../.../c.txt,c cc ccc cccc ccccc)
(hdfs:/.../.../d.txt,d dd ddd dddd ddddd)
(hdfs:/.../.../e.txt,e ee eee eeee eeeee)

After using this method, all that's left to do based on your question is to convert the result RDD to a DataFrame were each record will have its data in two columns (named by you): file_name and content .

+---------+-------------------+
|file_name|            content|
+---------+-------------------+
|    a.txt|a aa aaa aaaa aaaaa|
|    b.txt|b bb bbb bbbb bbbbb|
|    c.txt|c cc ccc cccc ccccc|
|    d.txt|d dd ddd dddd ddddd|
|    e.txt|e ee eee eeee eeeee|
+---------+-------------------+

This can happen with the snippet of code below

// create a scala spark context to use the wholeTextFiles method
val sc = spark.sparkContext

// create an RDD where the full path of each file is the key 
// and the file's content is the value,
// and get rid of the full path of the file
val input = sc.wholeTextFiles("hdfs://path/to/folder/*")
.map(file => (file._1.split('/').last, file._2))

// convert the RDD to a DataFrame and explicitly name the columns
val input_df = spark.createDataFrame(input).toDF("file_name", "content")