Google Dataflow API 按作业名称过滤

Question

Is there a way to filter Dataflow jobs by Job Name with REST API? I am looking for a way to get list of job details filter by job name. Currently, I am able to do it through cloud dataflow console, but not from dataflow rest api.

GET /v1b3/projects/{projectId}/jobs

Answer 1

The filter that is performed within the Dataflow console is not part of the API (It seems that the Dataflow API is requested to get the jobs but the frontend layer is the one that performs the filtering function).

Therefore, you could replicate this by following the same steps:

1- To list all jobs across all regions, use projects.jobs.aggregated (GET/v1b3/projects/{projectId}/jobs: aggregated). Additionally, this method allows you to pre-filter jobs for a specified job state .

Projects.jobs.list (GET/v1b3/projects/{projectId}/ jobs) is not recommended, as you can only get the list of jobs that are running in us-central1.

2- Both methods mentioned above, return a JSON ListJobsResponse object, this object contains a list of Jobs. Therefore, you can iterate this list in some programming language like Python and filter the jobs by a regex over the job name to get the desired jobs:

import json
import re
 
desired_name = 'REGEX_STRING'
 
filtered_jobs  = list()
 
with open('ListJobsResponse.json') as json_file:
   response_dict = json.load(json_file)
  
   jobs = response_dict['jobs']
 
   for j in jobs:
       x = re.search(desired_name, j['name'])
       if x:
           filtered_jobs.append(j)
 
print(filtered_jobs)

Answer 2

You can use filter the Dataflow jobs by name while using below API.

API Reference: https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs/aggregated

API Methods:

GET https://dataflow.googleapis.com/v1b3/projects/{projectId}/jobs:aggregated
or
GET https://dataflow.googleapis.com/v1b3/projects/{projectId}/jobs

parameters (works for both APIs):
    name: name of the dataflow job