Spark FileStreaming无法与foreachRDD一起使用

Question

I'm newbie to Spark, and i'm building a Small sample application which is a Spark fileStreaming one. All i wanted is to read the whole file in one go instead of reading line by line(i guess this is what the textFileStream does).

The code is below:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._

import org.apache.hadoop.io.LongWritable
import org.apache.hadoop.io.Text
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat

import scalax.io._

object SampleXML{

    def main(args: Array[String]){

        val logFile = "/home/akhld/mobi/spark-streaming/logs/sample.xml"

        val ssc = new StreamingContext("spark://localhost:7077","XML Streaming Job",Seconds(5),"/home/akhld/mobi/spark-streaming/spark-0.8.0-incubating",List("target/scala-2.9.3/simple-project_2.9.3-1.0.jar"))

        val lines = ssc.fileStream[LongWritable, Text, TextInputFormat]("/home/akhld/mobi/spark-streaming/logs/")

        lines.print()

        lines.foreachRDD(rdd => {
          rdd.count()  // prints counts

        })


        ssc.start()


    }
}

This code is failing with an exception saying that:

[error] /home/akhld/mobi/spark-streaming/samples/samplexml/src/main/scala/SampleXML.scala:31: value foreachRDD is not a member of org.apache.spark.streaming.DStream[(org.apache.hadoop.io.LongWritable, org.apache.hadoop.io.Text)]
[error]         ssc.fileStream[LongWritable, Text, TextInputFormat]("/home/akhld/mobi/spark-streaming/logs/").foreachRDD(rdd => {
[error]                                                                                                       ^
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 3 s, completed Feb 3, 2014 7:32:57 PM

If this is not the right way of displaying the contents of the files in the stream, Please help me with an example. I searched a lot but couldn't find the proper one to use fileStream.

Answer 1

Well, textFileStream in the Spark Streaming is more meant to continuously read and process files that are being written inside a directory. So if you have to process one file as a whole in one shot, it is simpler to use Spark directly!

 val lines = sparkContext.textFile(<file URL>)
 lines.foreach(line => println(line))

This will print all the lines in the file.

Answer 2

Also, I beleive the issue here is that you cannot call count() on an RDD inside a forEach stream. The reason is that, if you do, I think it blocks progress of the forEach block - and the stream consumer stops working.

I've created a JIRA for this https://issues.apache.org/jira/browse/SPARK-4040 .

I think there are some sensitive API calls you can make on RDDs when you are referencing them in a forEach block , but havent quite worked out all the details yet.