[英]How to configure flink jobs at runtime?
是否可以在運行時配置flink應用程序? 例如,我有一個流應用程序,該應用程序讀取輸入,進行一些轉換,然后過濾掉低於某個閾值的所有元素。 但是,我希望此閾值在運行時是可配置的,這意味着我可以更改此閾值而不必重新啟動flink作業。 示例代碼:
DataStream<MyModel> myModelDataStream = // get input ...
// do some stuff ...
.filter(new RichFilterFunction<MyModel>() {
@Override
public boolean filter(MyModel value) throws Exception {
return value.someValue() > someGlobalState.getThreshold();
}
})
// write to some sink ...
DataStream<MyConfig> myConfigDataStream = // get input ...
// ...
.process(new RichProcessFunction<MyConfig>() {
someGlobalState.setThreshold(MyConfig.getThreshold());
})
// ...
是否有可能實現這一目標? 例如,可以通過配置流更改的全局狀態。
是的,您可以使用BroadcastProcessFunction
做到這一點。 大概是這樣的:
MapStateDescriptor<Void, Threshold> bcStateDescriptor = new MapStateDescriptor<>(
"thresholds", Types.VOID, Threshold.class);
DataStream<MyModel> myModelDataStream = // get input ...
DataStream<Threshold> thresholds = // get input...
BroadcastStream<Threshold> controlStream = thresholds.broadcast(bcStateDescriptor);
DataStream<MyModel> result = myModelDataStream
.connect(controlStream)
.process(new MyFunction());
public class MyFunction extends BroadcastProcessFunction<MyModel, Long, MyModel> {
@Override
public void processBroadcastElement(Threshold newthreshold, Context ctx, Collector<MyModel> out) {
BroadcastState<Void, Threshold> bcState = ctx.getBroadcastState(new MapStateDescriptor<>("thresholds", Types.VOID, Threshold.class));
bcState.put(null, newthreshold);
}
@Override
public void processElement(MyModel model, Collector<MyModel> out) {
Threshold threshold = ctx.getBroadcastState(new MapStateDescriptor<>("threshold", Types.VOID, Threshold.class)).get(null);
if (threshold.value() == null || model.getData() > threshold.value()) {
out.collect(model);
}
}
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.