如何使用 Apache Beam 中的流输入 PCollection 请求 Redis 服务器？

Question

Using RedisIO , I'm trying to query a collection to a redis server.

The Redis server is ok and responding good only when batch pipeline (no streaming).

But, using streaming input data (from files) like this:

  PCollection<String> stream = pipeline.apply("ReadMyFile", TextIO.read().from("/home/out/**")
  .watchForNewFiles(Duration.standardSeconds(60), Watch.Growth.<String>never()))   
  .apply("ParseFn", ParDo.of(new ParseFn()))
  .apply("GlobalString", GlobalString.get(Duration.ZERO, Duration.standardSeconds(60)));

And then, apply the redisIO read() function:

 PCollection<KV<String, String>> redis = stream.apply(RedisIO.readAll().withEndpoint("127.0.0.1", 6379));

Finally, want to use the result collection, so:

 PCollection<String> result = redis.apply("Compose Final Object", ParDo.of(new DoFn<KV<String, String>, String>() {
    @ProcessElement
    public void processElement(ProcessContext c) {
      System.out.println(c.element().getKey());
      c.output(c.element().getKey());
    }
  }));

As far as I tested, the files are being loaded and processed as needed.

Answer 1

For further developments, it is a bug:

https://github.com/apache/beam/pull/10624/commits

https://issues.apache.org/jira/browse/BEAM-9134

It will be fixed soon.