I want to grab the dataflow's Job ID as a string before actually running the pipeline or inside of it.
As an example let's say I have this:
DataflowOptions options = optionsConst.getOptions();
Pipeline p = Pipeline.create(options);
String jobID = doSomthing(); <-- I'd like to grab the job id here
PCollection<KV<String, Object>> refData = p.apply("BigTable Access", BigtableIO.read()
.withProjectId(options.getBigtableProjectId())
.withInstanceId(options.getBigtableInstanceId())
.withTableId(options.getLookupTableId()))
.apply("Keep tags valid versions", ParDo.of(new MethodThatDoesSomethingWithJobID(**jobId**))); <-- Or inside this
Is not possible to do that, as the Job Id is created once you start to run it. What you can do is to save the output once you start run the pipeline, or to return the list job running
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.