如何使用 Apache Bean Java 获取具有多个 ParDo 的数据流中的管道状态

Question

Pipeline contain multiple ParDo functions(Refer below code).Need to send the Failed message to pubsub topic when Dataflow pipeline ParDo function failed. Tried with PipelineResults we are not able to get the status. Any centralized logic to implement status when dataflow pipeline failed? Kindly suggest me the idea to resolve the issue.

public class PubMessage {
        public static final Logger LOG = LoggerFactory.getLogger(PubMessage.class.getName());
    public static void main(String[] args) {
        PipelineOption options = PipelineOptionsFactory.fromArgs(args).as(PipelineOption.class);

        Pipeline pipeline = Pipeline.create(options);

        PCollection<String> input = pipeline.apply("Read Dummy File", new ReadDummyFile(options.getDummyFilepath()));

        Publish Pubsub Message to Topic
        input.apply("Pardo", ParDo.of(new msg(options.getPath()))).apply("Publish Pubsub Message",PubsubIO.writeMessages().to(options.getTopic()));

        // Publish second message to Topic      
        String print_topic = options.getTopic();
        input.apply("Pardo", ParDo.of(new msgSecond("Read text").apply("Publish Pubsub Message",PubsubIO.writeMessages().to(options.getTopic()));


        PipelineResult p = pipeline.run();
    if (PipelineResult.State.FAILED.equals(p.waitUntilFinish())) {
        throw new RuntimeException("Pipeline failed for unknown reason");
        // send pubsub msg
    }

msg

public class msg extends DoFn<String, PubsubMessage> {
    @ProcessElement
    public void processElement(ProcessContext c) {
        //get the value sending msg to topic
        c.output(message);
    }
}   
    
msgSecond
    
   public class msgSecondextends DoFn<String, PubsubMessage> {
        @ProcessElement
        public void processElement(ProcessContext c) {
            //get the value sending msg to topic
            c.output(message);
        }
    }

          
    
    

Answer 1

A streaming pipeline will keep trying rather than completing in a failed state. You could publish the message from within the DoFn itself.