如何使用 impersonate_service_account 进行流式光束管道

Question

how can I run beam pipeline locally ( DirectRunner ) in streaming mode using impersonate_service_account credentials? (in python3)

I want to get rid of Service Account key file ( GOOGLE_APPLICATION_CREDENTIALS environment variable)

here's minimal code

options = PipelineOptions()
options.view_as(GoogleCloudOptions).impersonate_service_account = <MY SERVICE ACCOUNT EMAIL>
p = beam.Pipeline(options=options)
source = pubsub.ReadFromPubSub(subscription=SUB,
                               with_attributes=True,
                               timestamp_attribute = "evt_time")

where can I set source credentials (my own GCP user account credentials set via gcloud auth application-default login ) i checked source code for PipelineOptions(), ReadFromPubSub() - none seem to have suitable way to inject source credentials.

thank you for advice.

Thank you for your advice?

Answer 1

If you just want to avoid exporting a key file, you can use a User principal and configure your gcloud SDK with gcloud auth application-default login .

This command is useful when you are developing code that would normally use a service account but need to run the code in a local development environment where it's easier to provide user credentials. The credentials will apply to all API calls that make use of the Application Default Credentials client library. Do not set the GOOGLE_APPLICATION_CREDENTIALS environment variable if you want to use the credentials generated by this command in your local development.

Or if you are trying to create an impersonating chain, you can use comma separated SAs in the impersonate_service_account pipeline option {delegated_principal},{targeted_principal} . The auth logic that extracts the information is here

Answer 2

Unfortunately, not all Google Cloud services support ADC impersonated_service_account .

If you are not running on a Google Cloud compute service, you will need credentials on the system. Using the CLI gcloud just masks the fact that credentials are still stored on the system. The CLI is for developer convenience and not for security. You are trying to switch from GOOGLE_APPLICATION_CREDENTIALS to another method that requires storing different credentials. That will not improve your security .

Even if you used OAuth User credentials and then switched to impersonated credentials (I have written articles on that), you still store credentials and secrets on the local system. Basically, you have a chicken and egg situation. You need credentials to impersonate other credentials . When running in the cloud, those credentials are managed for you. Outside the cloud, you manage the credentials.