[英]Stop spark streaming
I want to stop java streaming context in spark after processing 100 records in a file. 我想在处理文件中的100条记录后停止Spark中的Java流上下文。 The problem is the code in if statement does not executed when streaming starts. 问题是流开始时if语句中的代码未执行。 The following code will explain my idea: 以下代码将解释我的想法:
public static void main(String[] args) throws Exception {
int ff = testSparkStreaming();
System.out.println("wqwqwq");
System.out.println(ff);
}
public static int testSparkStreaming() throws IOException, InterruptedException {
int numberInst = 0
String savePath = "Path to Model";
final NaiveBayesModel savedModel = NaiveBayesModel.load(jssc.sparkContext().sc(), savePath);
BufferedReader br = new BufferedReader(new FileReader("C://testStream//copy.csv"));
Queue<JavaRDD<String>> rddQueue = new LinkedList<JavaRDD<String>>();
List<String> list = Lists.newArrayList();
String line = "";
while ((line = br.readLine()) != null) {
list.add(line);
}
br.close();
rddQueue.add(jssc.sparkContext().parallelize(list));
numberInst+= list.size();
JavaDStream<String> dataStream = jssc.queueStream(rddQueue);
dataStream.print();
if (numberInst == 100){
System.out.println("should stop");
jssc.wait();
}
jssc.start();
jssc.awaitTermination();
return numberInst;
}
My question is how can I stop the streaming when numberInst == 100 and move the execution to main method to run the following statements. 我的问题是,当numberInst == 100时如何停止流传输并将执行移至main方法以运行以下语句。
PS: in the previous code, If statement is not executed: PS:在前面的代码中,如果未执行If语句:
if (numberInst == 100){
System.out.println("should stop");
jssc.wait();
}
You can try this: 您可以尝试以下方法:
jssc.start();
while (numberInst < 100){
jssc.awaitTerminationOrTimeout(1000); // 1 second polling time, you can change it as per your usecase
}
jssc.stop();
您是否尝试过像Thread那样停止操作,我是说Interrup。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.