[英]How to access Java Spark Broadcast variable?
我正在嘗試在 spark 中廣播Dataset
,以便從map
函數中訪問它。 第一個 print 語句按預期返回廣播數據集的第一行。 不幸的是,第二個 print 語句沒有返回結果。 此時執行只是掛起。 知道我做錯了什么嗎?
Broadcast<JavaRDD<Row>> broadcastedTrainingData = this.javaSparkContext.broadcast(trainingData.toJavaRDD());
System.out.println("Data:" + broadcastedTrainingData.value().first());
JavaRDD<Row> rowRDD = this.javaSparkContext.parallelize(stringAsList).map((Integer row) -> {
System.out.println("Data (map):" + broadcastedTrainingData.value().first());
return RowFactory.create(row);
});
以下偽代碼突出了我想要實現的目標。 我的主要目標是廣播訓練數據集,這樣我就可以在地圖函數中使用它。
public Dataset<Row> getWSSE(Dataset<Row> trainingData, int clusterRange) {
StructType structType = new StructType();
structType = structType.add("ClusterAm", DataTypes.IntegerType, false);
structType = structType.add("Cost", DataTypes.DoubleType, false);
List<Integer> stringAsList = new ArrayList<>();
for (int clusterAm = 2; clusterAm < clusterRange + 2; clusterAm++) {
stringAsList.add(clusterAm);
}
Broadcast<Dataset> broadcastedTrainingData = this.javaSparkContext.broadcast(trainingData);
System.out.println("Data:" + broadcastedTrainingData.value().first());
JavaRDD<Row> rowRDD = this.javaSparkContext.parallelize(stringAsList).map((Integer row) -> RowFactory.create(row));
StructType schema = DataTypes.createStructType(new StructField[]{DataTypes.createStructField("ClusterAm", DataTypes.IntegerType, false)});
Dataset wsse = sqlContext.createDataFrame(rowRDD, schema).toDF();
wsse.show();
ExpressionEncoder<Row> encoder = RowEncoder.apply(structType);
Dataset result = wsse.map(
(MapFunction<Row, Row>) row -> RowFactory.create(row.getAs("ClusterAm"), new KMeans().setK(row.getAs("ClusterAm")).setSeed(1L).fit(broadcastedTrainingData.value()).computeCost(broadcastedTrainingData.value())),
encoder);
result.show();
broadcastedTrainingData.destroy();
return wsse;
}
DataSet<Row> trainingData = ...<Your dataset>;
//Creating the broadcast variable. No need to write classTag code by hand
// use akka.japi.Util which is available
Broadcast<Dataset<Row>> broadcastedTrainingData = spark.sparkContext()
.broadcast(trainingData, akka.japi.Util.classTag(DataSet.class));
//Here is the catch.When you are iterating over a Dataset,
//Spark will actally run it in distributed mode. So if you try to accees
//Your object directly (e.g. trainingData) it would be null .
//Cause you didn't ask spark to explicitly send tha outside variable to
//each machine where you are running this for each parallelly.
//So you need to use Broadcast variable.(Most common use of Broadcast)
someSparkDataSet.foreach((row) -> {
DataSet<Row> recieveBrdcast = broadcastedTrainingData.value();
...
...
})
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.