In Flink, how to write DataStream to single file?

Question

The writeAsText or writeAsCsv methods of a DataStream write as many files as worker threads. As far as I could see, the methods only let you specify the path to these files and some formatting.

For debugging and testing purposes, it would be really useful to be able to print everything to a single file, without having to change the set up to having a single worker thread.

Is there any non-overly-complicated way to achieve this? I suspect it should be possible implementing a custom SinkFunction , but not sure about that one (besides, it also feels like a hassle for something that seems relatively simple).

Answer 1

您可以通过将并行度设置为1来实现此目的。这样，写入仅在一台计算机上进行。

writeAsText(path).setParallelism(1);