Is it possible to update existing Google cloud dataflow pipelines when using template for deployment ?

Question

When deploying google dataflow pipeline as templates, Is it possible to update the pipeline using another version of the template?

Basically, I am looking for a combinationhttps://cloud.google.com/dataflow/pipelines/updating-a-pipeline with https://cloud.google.com/dataflow/docs/templates/overview

Answer 1

The feature of updating an existing job from template API is not ready yet. (We are working on it).

At the time, you can probably make use our public repo (basically the source code for these templates) to do it. Basically, you can just build and launch the job to "update" the running job from a shell.

https://github.com/GoogleCloudPlatform/DataflowTemplates

Answer 2

Now it's possible to update a template streaming Dataflow job.

Using the REST API, set the update parameter to true:

POST https://dataflow.googleapis.com/v1b3/projects/YOUR_PROJECT_ID/locations/LOCATION/templates:launch?gcsPath=gs://YOUR_BUCKET_NAME/templates/TemplateName
{
    "jobName": "JOB_NAME",
    "parameters": {
        "topic": "projects/YOUR_PROJECT_ID/topics/YOUR_TOPIC_NAME",
        "table": "YOUR_PROJECT_ID:YOUR_DATASET.YOUR_TABLE_NAME"
    },
    "environment": {
        "tempLocation": "gs://YOUR_BUCKET_NAME/temp",
        "zone": "us-central1-f"
    }
    "update": true
}

The update option is not present usinggcloud dataflow jobs run .

https://cloud.google.com/dataflow/docs/guides/templates/running-templates#example-3:-updating-a-custom-template-streaming-job https://cloud.google.com/dataflow/docs/reference/rest/v1b3/LaunchTemplateParameters