Is there any way to do processing after GCP dataflow has completed the job using apache beam?

Question

Is there any way to do post processing after the GCP dataflow status is done/completed. I have process where dataflow reads a file from GCP storage in batches and does some external api calls for transformation and writes back to another file. I need to do some extra processing after all the batches are transformed/processed. Is there any way to do that ? I am using Apache Beam and template to run GCP datatflow.

Answer 1

Yes. There are two options to mention:

1. You can use Cloud Composer to orchestrate a Dataflow job with other actions before and after. See this article .
2. If you launch your job via code, you can call PipelineResult.waitUntilFinish() and then run additional code afterwards.