Provide BigQuery credentials in Apache-Beam pipeline coded in python

Question

I'm trying to read data from bigquery in my beam pipeline using cloud dataflow runner. I want to provide a credentials to access the project.

I've seen examples in Java but none in Python.

The only possibility I found is to use the: --service_account_email argument But what if I want to give the.json key information in the code itself in all the options like: google_cloud_options.service_account = '/path/to/credential.json'

options = PipelineOptions(flags=argv)
google_cloud_options = options.view_as(GoogleCloudOptions)
google_cloud_options.project = 'project_name'
google_cloud_options.job_name = 'job_name'
google_cloud_options.staging_location = 'gs://bucket'
google_cloud_options.temp_location = 'gs://bucket'
options.view_as(StandardOptions).runner = 'DataflowRunner'

with beam.Pipeline(options=options) as pipeline:
    query = open('query.sql', 'r')
    bq_source = beam.io.BigQuerySource(query=query.read(), use_standard_sql=True)
    main_table = \
        pipeline \
        | 'ReadAccountViewAll' >> beam.io.Read(bq_source) \

Java has a method getGcpCredential but cant find one in Python...

Any ideas?

Answer 1

The --service_account_email is the recommended approach as mentioned here . Downloading the key and storing it locally or on GCE is not recommended.

Answer 2

For the cases where it is required to use a different path for the json file within the code, you can try the following python Authentication workarounds:

client = Client.from_service_account_json('/path/to/keyfile.json')

or

client = Client(credentials=credentials)

Here is an example for creating custom credentials from a file:

credentials = service_account.Credentials.from_service_account_file(
    key_path,
    scopes=["https://www.googleapis.com/auth/cloud-platform"],
)