Google Cloud Dataflow - providing an sdk_location in pipeline options

Question

background:

I have a web app that calls google dataflow and recently wanted to use the sdk_location parameter in the pipeline options.
I downloaded the apache-beam sdk and uploaded this to a gcs bucket as a tar.gz file.
I then added sdk_location={location of the.tar.gz file}

However when I now make the dataflow API call, I get the following error in the worker startup logs.
Failed to install worker package.

Has anyone else had this issue?
And is that the correct tarball to have provided?

Answer 1

When using Dataflow, you don't have to use the sdk_location option unless you make changes to the Beam SDK itself.

If it fails to install worker package, check whether you are missing necessary dependencies to run your job. For example, if you are using the Python SDK: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/