Spark Streaming仅流在流初始化时间之后创建的流文件

Question

Is there any way to configure the textFileStream source such that it will process any file added to the source directory regardless of the file create time?

To demonstrate the issue, I created a basic Spark Streaming application that uses textFileStream as a source and prints the stream contents to the console. When an existing file created prior to running the application is copied into the source directory, nothing is printed to the console. When a file created after the application starts running is copied to the source directory, the file contents are printed. Below is my code for reference.

val conf = new SparkConf().setAppName("Streaming Test")
                          .setMaster("local[*]")

val spark = new SparkContext(conf)
val ssc = new StreamingContext(spark, Seconds(5))

val fileStream = ssc.textFileStream("/stream-source")

val streamContents = fileStream.flatMap(_.split(" "))

streamContents.print()

Answer 1

This is the documented behavior of the FileInputDStream .

If we would like to consume existing files in that directory, we can use the Spark API to load these files and apply our desired logic to them.

val existingFiles = sparkContext.textFile(path)

or

val existingFilesDS = sparkSession.read.text(path)

And then after, setup and start the streaming logic. We could even use the data of the already existing files in the processing of the new ones.