多个火花上下文错误

Question

I have this spark code below:

    import org.apache.hadoop.hbase.client._
    import org.apache.hadoop.hbase.{ HBaseConfiguration, HTableDescriptor }
    import org.apache.hadoop.hbase.mapreduce.TableInputFormat
    import org.apache.hadoop.hbase.io.ImmutableBytesWritable
    import org.apache.hadoop.hbase.util.Bytes

    import kafka.serializer.StringDecoder

    import org.apache.spark._
    import org.apache.spark.SparkContext._
    import org.apache.spark.streaming._
    import org.apache.spark.streaming.kafka._

object Hbase {
    def main(args: Array[String]) {
          val sparkConf = new SparkConf().setAppName("Spark-Hbase").setMaster("local[2]")
          val sc = new SparkContext(sparkConf)

          ...

          val ssc = new StreamingContext(sparkConf, Seconds(3))
          val kafkaBrokers = Map("metadata.broker.list" -> "localhost:9092")
          val topics = List("test").toSet
          val lines = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaBrokers, topics)
    }
}

Now the error I am getting is:

Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true.

Is there anything wrong with my code above? I do not see where I am creating the context again...

Answer 1

These are the two SparkContext you're creating. This is not allowed.

val sc = new SparkContext(sparkConf)
val ssc = new StreamingContext(sparkConf, Seconds(3))

You should create the streaming context from the original context.

val ssc = new StreamingContext(sc, Seconds(3))

Answer 2

you are initializing two spark context in the same JVM ie (sparkContext and streamingContext). That's why you are getting this exception. you can set spark.driver.allowMultipleContexts = true in config. Although, multiple Spark contexts is discouraged. You can get unexpected results.