[英]Apache Flink (Error in stdout in cluster)
在Apache Flink
,我無法在std out
看到輸出,但我的工作正在成功運行並且數據即將到來
當您在集群上運行您的作業時,DataStreams 被打印到 TaskManager 進程的標准輸出。 這個 TaskManager stdout 被定向到 Flink 根目錄的 ./log/ 目錄中的一個 .out 文件。 我相信這是在這里您已經看到了您的輸出。
我不知道是否可以更改 TaskManagers 的標准輸出,但是,一個快速而骯臟的解決方案可能是將輸出寫入套接字:
output.writeToSocket(outputHost, outputPort, new SimpleStringSchema())
public static void main(String[] args) throws Exception {
// the host and the port to connect to
final String hostname = "192.168.1.73";
final int port = 9000;
final StreamExecutionEnvironment env = StreamExecutionEnvironment.createRemoteEnvironment("192.168.1.68", 6123);
// get input data by connecting to the socket
DataStream<String> text = env.socketTextStream(hostname, port, "\n");
// parse the data, group it, window it, and aggregate the counts
DataStream<WordWithCount> windowCounts = text
.flatMap(new FlatMapFunction<String, WordWithCount>() {
public void flatMap(String value, Collector<WordWithCount> out) {
for (String word : value.split("\\s")) {
out.collect(new WordWithCount(word, 1L));
}
}
})
.keyBy("word").timeWindow(Time.seconds(5))
.reduce(new ReduceFunction<WordWithCount>() {
public WordWithCount reduce(WordWithCount a, WordWithCount b) {
return new WordWithCount(a.word, a.count + b.count);
}
});
// print the results with a single thread, rather than in parallel
windowCounts.print().setParallelism(1);
env.execute("Socket Window WordCount");
}
public static class WordWithCount {
public String word;
public long count;
public WordWithCount() {
}
public WordWithCount(String word, long count) {
this.word = word;
this.count = count;
}
@Override
public String toString() {
return word + " : " + count;
}
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.