无法在转换中访问广播变量

Question

我在转换函数中访问变量时遇到问题。 有人可以帮帮我吗？ 这是我的相关课程和功能。

@SerialVersionUID(889949215L)
object MyCache extends Serializable {
    @transient lazy val logger = Logger(getClass.getName)
    @volatile var cache: Broadcast[Map[UUID, Definition]] = null

    def getInstance(sparkContext: SparkContext) : Broadcast[Map[UUID, Definition]] = {
        if (cache == null) {
            synchronized {
                val map = sparkContext.cassandraTable("keyspace", "table")
                   .collect()
                   .map(m => m.getUUID("id") ->
                        Definition(m.getString("c1"), m.getString("c2"), m.getString("c3"),
                                m.getString("c4"))).toMap
                cache = sparkContext.broadcast(map)
            }
        }
        cache
    }
}

在另一个文件中：

object Processor extends Serializable {
    @transient lazy val logger = Logger(getClass.getName)

    def processData[T: ClassTag](rawStream: DStream[(String, String)], ssc: StreamingContext,
                                        processor: (String, Broadcast[Map[UUID, Definition]]) => T): DStream[T] = {
        MYCache.getInstance(ssc.sparkContext)
        var newCacheValues = Map[UUID, Definition]()
        rawStream.cache()
        rawStream
          .transform(rdd => {
                val array = rdd.collect()
                array.foreach(r => {
                      val value = getNewCacheValue(r._2, rdd.context)
                      if (value.isDefined) {
                          newCacheValues = newCacheValues + value.get
                      }
                })
                rdd
           })
       if (newCacheValues.nonEmpty) {
           logger.info(s"Rebroadcasting.  There are ${newCacheValues.size} new values")
           logger.info("Destroying old cache")
           MyCache.cache.destroy()
           // this is probably wrong here, destroying object, but then referencing it.  But I haven't gotten to this part yet.
           MyCache.cache = ssc.sparkContext.broadcast(MyCache.cache.value ++ newCacheValues)
       }
       rawStream
          .map(r => {
               println("######################")
               println(MyCache.cache.value)
               r
          })
          .map(r => processor(r._2, MyCache.cache.value))
          .filter(r => null != r)
   }
}

每次运行时我都会SparkException: Failed to get broadcast_1_piece0 of broadcast_1尝试访问cache.value时SparkException: Failed to get broadcast_1_piece0 of broadcast_1

当我添加println(MyCache.cache.values)后右.getInstance我能够收听广播的变量，但是当我将其部署到mesos簇我无法再次访问该广播值，但有空指针异常。

更新：

我看到的错误是在println(MyCache.cache.value) 。 我不应该添加这个包含destroy的if语句，因为我的测试永远不会发生这种情况。

我的应用程序的基础是，我有一个cassandra表，不会更新。 但我需要对某些流数据进行一些验证。 所以我想将这个表中的所有数据（不是更新的数据）提取到内存中。 getInstance在启动时拉入整个表，然后检查我的所有流数据，看看我是否需要再次从cassandra中取出（我将很少这样做）。 转换和收集是我检查是否需要提取新数据的地方。但由于我的表有可能会更新，我需要偶尔更新广播。 所以我的想法是摧毁它然后重播。 一旦我得到其他工作，我会更新。

如果我注释掉destroy和重播，我会得到同样的错误。

另一个更新：

我需要在processor访问广播变量这行： .map(r => processor(r._2, MyCache.cache.value)) 。

我能够在变换中广播变量，如果我在变换中执行println(MyCache.cache.value) ，那么我的所有测试都会通过，然后我就可以在processor访问广播了

更新：

rawStream
    .map(r => {
      println("$$$$$$$$$$$$$$$$$$$")
      println(metrics.value)
      r
    })

这是我遇到这一行时得到的堆栈跟踪。

    ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 135.0 (TID 114)
    java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_1_piece0 of broadcast_1
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1222)
        at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
        at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
        at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
        at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
        at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
        at com.uptake.readings.ingestion.StreamProcessors$$anonfun$processIncomingKafkaData$4.apply(StreamProcessors.scala:160)
        at com.uptake.readings.ingestion.StreamProcessors$$anonfun$processIncomingKafkaData$4.apply(StreamProcessors.scala:158)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
        at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:414)
        at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)
        at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
        at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
    Caused by: org.apache.spark.SparkException: Failed to get broadcast_1_piece0 of broadcast_1
        at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
        at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:137)
        at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
        at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:120)
        at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:175)
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1219)
        ... 24 more

Answer 1

[更新的答案]

你得到一个错误，因为rawStream.map的代码即MyCache.cache.value正在其中一个执行程序上执行，并且MyCache.cache仍然为null ！

当您执行MyCache.getInstance ，它在驱动程序上创建了MyCache.cache值并将其广播好。 但是你并没有在map方法中引用相同的对象，因此它不会被发送到执行程序。 相反，由于您直接引用MyCache ，执行MyCache.cache在自己的MyCache对象副本上调用MyCache.cache ，这显然是null。

您可以通过首先在驱动程序中获取cache广播对象的实例并在地图中使用该对象来使其按预期工作。 以下代码应该适合您 -

val cache = MYCache.getInstance(ssc.sparkContext)
rawStream.map(r => {
                     println(cache.value)
                     r
             })