简体   繁体   中英

How do I get a dataflow Job ID from within the pipeline or before the pipeline ran?

I want to grab the dataflow's Job ID as a string before actually running the pipeline or inside of it.

As an example let's say I have this:

DataflowOptions options = optionsConst.getOptions();         
Pipeline p = Pipeline.create(options);
String jobID = doSomthing(); <-- I'd like to grab the job id here

PCollection<KV<String, Object>> refData = p.apply("BigTable Access", BigtableIO.read()
                .withProjectId(options.getBigtableProjectId())
                .withInstanceId(options.getBigtableInstanceId())
                .withTableId(options.getLookupTableId()))
                .apply("Keep tags valid versions", ParDo.of(new MethodThatDoesSomethingWithJobID(**jobId**))); <-- Or inside this

Is not possible to do that, as the Job Id is created once you start to run it. What you can do is to save the output once you start run the pipeline, or to return the list job running

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM