How scala object member work with spark rdd

Question

I have a spark application, that output result to redis.

It works fine on local mode, but cannot connect the redisHost with the args(0) that I assign like 10.242.10.100 on yarn-cluster mode.

The redisHost is unchanged 127.0.0.1 .

object TestSparkClosure {
  val logger: Logger = LoggerFactory.getLogger(TestSparkClosure.getClass)
  var redisHost = "127.0.0.1"
  var redisPort = 6379

  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("TestSparkClosure")

    if (args.length > 0) {
      redisHost = args(0)
    } else {
      conf.setMaster("local")
    }
    val sparkContext = new SparkContext(conf)
    var rdd = getRdd(sparkContext)
    rdd.foreachPartition(partitionOfRecords => {
      logger.info("host:port:" + redisHost + ":" + redisPort.toString)
      val jedis = new Jedis(redisHost, redisPort)
      partitionOfRecords.foreach(pair => {
        val keystr = pair._1
        val valuestr = pair._2
        jedis.set(keystr, valuestr)
      })
    })
  }

  def getRdd(spark: SparkContext): RDD[(String, String)] = {
    val rdd = spark.parallelize(List("2017\t1", "2018\t2", "2017\t3", "2018\t4", "2017\t5", "2018\t6")).map(line => {
      val cols = line.split("\t")
      (cols(0), cols(1))
    })
    rdd.reduceByKey((x, y) => {
      ((x.toInt + y.toInt).toString)
    }, 3)
  }
}

When I replace redisHost with local variable like this, It works fine again.

    var localRedisHost = redisHost
    rdd.foreachPartition(partitionOfRecords => {
      logger.info("host:port:" + localRedisHost + ":" + redisPort.toString)
      val jedis = new Jedis(localRedisHost , redisPort)
      partitionOfRecords.foreach(pair => {
        val keystr = pair._1
        val valuestr = pair._2
        jedis.set(keystr, valuestr)
      })
    })

Can anyone explain how the spark closure work here?

Thanks so much.

Answer 1

Its because you are using a variable which isnt able to use serialization. when you define a local element it can so you are able to use it inside of the RDD.