[英]How do I time the checkpointing in Apache Flink streaming?
我正在运行 Apache Flink 的欺诈检测器示例,并将 RocksDB 作为我的 state 后端。 我想知道 Apache Flink 检查点 state 需要多长时间。
我的方法是在检查点功能之前和之后打印时间。
我找不到检查点 state 的函数/类或任何代码片段,我尝试通过 IDE 进行调试但徒劳无功。
这是我到目前为止所经历的:
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package spendreport;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.walkthrough.common.sink.AlertSink;
import org.apache.flink.walkthrough.common.entity.Alert;
import org.apache.flink.walkthrough.common.entity.Transaction;
import org.apache.flink.walkthrough.common.source.TransactionSource;
//org.apache.flink.contrib.streaming.state
import org.apache.flink.contrib.streaming.state.RocksDBStateBackend;
import javax.security.auth.login.Configuration;
/**
* Skeleton code for the datastream walkthrough
*/
public class FraudDetectionJob {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// env.setStateBackend(new RocksDBStateBackend(filebackend, true));
// Enabling Checkpoint
long checkpointInterval = 5000;
env.enableCheckpointing(checkpointInterval);
// Enable Web UI
// Configuration conf = new Configuration();
// env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
DataStream<Transaction> transactions = env
.addSource(new TransactionSource())
.name("transactions");
DataStream<Alert> alerts = transactions
.keyBy(Transaction::getAccountId)
.process(new FraudDetector())
.name("fraud-detector");
alerts
.addSink(new AlertSink())
.name("send-alerts");
env.execute("Fraud Detection");
}
}
我步入execute
function 并找到代码设置检查点配置的许多地方(例如检查超时间隔等)。 但是,我找不到实际检查执行检查点的 function。
自己衡量这一点并不容易,因为检查点部分在运行用户函数的线程中完成,部分在另一个线程中异步完成。
获取有关检查点影响的一些信息的最佳方法是查看检查点指标,这些指标被方便地收集在一起并显示在 web UI 中:
通常,当您在 IDE 中运行时,web UI 不可用,但您可以更改此设置:
Configuration conf = new Configuration();
env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
为此,您还必须添加此依赖项:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime-web_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.