将数据集应用为Spark中的广播

Question

I have two datasets, and i need to register one dataset(smaller one) as Broadcast when trying to register i am anable to use broadcast functions.

Here is the code:

JavaRDD<String> maps = ctx.textFile("C:\\Users\\sateesh\\Desktop\\country.txt");
Broadcast<JavaRDD<String>> broadcastVar = ctx.broadcast(maps);
//Broadcast<Map<Integer, String>> broadcastVar = ctx.broadcast(map);
List<Integer> list = new ArrayList<Integer>();
list.add(1);
list.add(2);
list.add(9);
JavaRDD<Integer> listrdd = ctx.parallelize(list);
JavaRDD<Object> mapr = listrdd.map(x -> broadcastVar.value());
System.out.println(mapr.collect());

Here I am not able to get broadcastVar.value().get(x) . If iregister any manual map as broadcast its works well, but in case of text files it doesn't works.

Answer 1

In order to broadcast any data to cluster it has to be from driver . So, collect() your rdd and broadcast it.

JavaRDD<String> rdd = ctx.textFile("C:\\Users\\sateesh\\Desktop\\country.txt");

Broadcast<List<String>> broadcastVar = ctx.broadcast(rdd.collect());

Please be aware collect() will bring entire rdd to driver it might throw OOM exception. Broadcast is suggested for less sized data.