如何在谷歌数据流中获取现有作业的“templateLocation”参数值

Question

I have a list of an existing jobs running in a google dataflow. I would like to list the jobs running for last x number of days and would like to recycle those programmatically. To achieve this I need the name of the template used for a particular job. We can easily get this information from Console in a Job Info view. However i would like to know if there is any way to get this info from Gcloud command or from a API.

Your early response will be appreciated.

Thanks Sarang

Answer 1

- Solution 1:

You can use GCloud sdk and a shell script to achieve your need:

https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/list

Filter jobs with the given name:

gcloud dataflow jobs list --filter="name=my-wordcount"

List jobs with from a given region:

gcloud dataflow jobs list --region="europe-west1"

List jobs created this year:

gcloud dataflow jobs list --created-after=2018-01-01

List jobs created more than a week ago:

gcloud dataflow jobs list --created-before=-P1W

Many filters and parameters are proposed to apply your use case.

- Solution 2

You can use the rest api for Dataflow jobs:

https://cloud.google.com/dataflow/docs/reference/rest

Example:

GET /v1b3/projects/{projectId}/locations/{location}/jobs
List the jobs of a project.

Answer 2

There is no direct way of getting the template location and name. To achieve above requirement I have defined the pattern for the template name and job name and based on the defined pattern, while recycling the job, template name has been computed and passed on to the API call.