[英]Java Spark flatMap seems to be losing items in ArrayList
我正在使用spark / cassandra驱动程序在cassandra中遍历数十亿行,并提取数据以运行统计信息。 为此,我在每行数据上运行一个FOR
循环,如果它属于一类数据的标准,我称之为“通道”,那么我将其以K,V对形式添加到ArrayList中。渠道,力量。
[[Channel,Power]]
基于for循环的迭代增量,通道应为静态。 例如,如果我的频道范围是0到10(增量为2),那么频道将是0、2、4、6、8、10
FOR
循环在当前数据行上运行,并检查数据是否落在通道内,如果是,则以[[Channel,Power]]的格式将其添加到ArrayList Data中
然后前进到下一行并执行相同的操作。 一旦遍历所有行,它将递增到下一个通道并重复该过程。
问题是有数十亿行符合同一通道的条件,所以我不确定是否应该使用ArrayList
和flatMap
或其他功能,因为每次运行它的结果都会略有不同,并且通道不是静态的像他们应该的那样。
一小部分数据[[Channel,Power]]将是:
[[2,5]]
[[2,10]]
[[2,5]]
[[2,15]]
[[2,5]]
请注意,由于我在每个这些频道上运行min,max,average统计信息,因此需要保留一些重复项。
频道2:最低5,最高15,平均8
我的代码如下:
JavaRDD<MeasuredValue> rdd = javaFunctions(sc).cassandraTable("SparkTestB", "Measured_Value", mapRowTo )
.select("Start_Frequency","Bandwidth","Power");
JavaRDD<Value> valueRdd = rdd.flatMap(new FlatMapFunction<MeasuredValue, Value>(){
@Override
public Iterable<Value> call(MeasuredValue row) throws Exception {
long start_frequency = row.getStart_frequency();
float power = row.getPower();
long bandwidth = row.getBandwidth();
// Define Variable
long channel,channel_end, increment;
// Initialize Variables
channel_end = 10;
increment = 2;
List<Value> list = new ArrayList<>();
// Create Channel Power Buckets
for(channel = 0; channel <= channel_end; ){
if( (channel >= start_frequency) && (channel <= (start_frequency + bandwidth)) ) {
list.add(new Value(channel, power));
} // end if
channel+=increment;
} // end for
return list;
}
});
sqlContext.createDataFrame(valueRdd, Value.class).groupBy(col("channel"))
.agg(min("power"), max("power"), avg("power"))
.write().mode(SaveMode.Append)
.option("table", "results")
.option("keyspace", "model")
.format("org.apache.spark.sql.cassandra").save();
我的课程是对反射的追随:
public class Value implements Serializable {
public Value(Long channel, Float power) {
this.channel = channel;
this.power = power;
}
Long channel;
Float power;
public void setChannel(Long channel) {
this.channel = channel;
}
public void setPower(Float power) {
this.power = power;
}
public Long getChannel() {
return channel;
}
public Float getPower() {
return power;
}
@Override
public String toString() {
return "[" +channel +","+power+"]";
}
}
public static class MeasuredValue implements Serializable {
public MeasuredValue() { }
public long start_frequency;
public long getStart_frequency() { return start_frequency; }
public void setStart_frequency(long start_frequency) { this.start_frequency = start_frequency; }
public long bandwidth ;
public long getBandwidth() { return bandwidth; }
public void setBandwidth(long bandwidth) { this.bandwidth = bandwidth; }
public float power;
public float getPower() { return power; }
public void setPower(float power) { this.power = power; }
}
我发现差异与我的频道化算法有关。 我用以下替换来解决问题。
// Create Channel Power Buckets
for(; channel <= channel_end; channel+=increment ){
//Initial Bucket
while((start_frequency >= channel) && (start_frequency < (channel + increment))){
list.add(new Value(channel, power));
channel+=increment;
}
//Buckets to Accomodate for Bandwidth
while ((channel <= channel_end) && (channel >= start_frequency) && (start_frequency + bandwidth) >= channel){
list.add(new Value(channel, power));
channel+=increment;
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.