Google Cloud Composer: How to trigger a DAG Run from a local dev machine?

Question

Background

I have a Google Cloud Composer 1 environment running on GCP.

I have written a Google Cloud Function that, when run in the cloud, successfully triggers a DAG Run in my Composer environment. My code is based on and almost identical to the code in the Trigger DAGs with Cloud Functions guide from GCP documentation.

Here is the section of code most relevant to my question ( source ):

from google.auth.transport.requests import Request
from google.oauth2 import id_token
import requests


def make_iap_request(url, client_id, method='GET', **kwargs):
    """Makes a request to an application protected by Identity-Aware Proxy.
    Args:
      url: The Identity-Aware Proxy-protected URL to fetch.
      client_id: The client ID used by Identity-Aware Proxy.
      method: The request method to use
              ('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE')
      **kwargs: Any of the parameters defined for the request function:
                https://github.com/requests/requests/blob/master/requests/api.py
                If no timeout is provided, it is set to 90 by default.
    Returns:
      The page body, or raises an exception if the page couldn't be retrieved.
    """
    # Set the default timeout, if missing
    if 'timeout' not in kwargs:
        kwargs['timeout'] = 90

    # Obtain an OpenID Connect (OIDC) token from metadata server or using service
    # account.
    open_id_connect_token = id_token.fetch_id_token(Request(), client_id)

    # Fetch the Identity-Aware Proxy-protected URL, including an
    # Authorization header containing "Bearer " followed by a
    # Google-issued OpenID Connect token for the service account.
    resp = requests.request(
        method, url,
        headers={'Authorization': 'Bearer {}'.format(
            open_id_connect_token)}, **kwargs)
    if resp.status_code == 403:
        raise Exception('Service account does not have permission to '
                        'access the IAP-protected application.')
    elif resp.status_code != 200:
        raise Exception(
            'Bad response from application: {!r} / {!r} / {!r}'.format(
                resp.status_code, resp.headers, resp.text))
    else:
        return resp.text

Challenge

I want to be able to run the same Cloud Function locally on my dev machine. When I try to do that, the function crashes with this error message:

google.auth.exceptions.DefaultCredentialsError: Neither metadata server or valid service account credentials are found.

This makes sense because the line that throws the error is:

    google_open_id_connect_token = id_token.fetch_id_token(Request(), client_id)

Indeed when running locally the Metadata Server is not available and I don't know how to make valid service account credentials available to the call to fetch_id_token() .

Question

My question is - What do I need to change in order to be able to securely obtain the OpenID Connect token when I run my function locally?

Answer 1

I've been able to run my code locally without changing it. Below are the details though I'm not sure this is the most secure option to get it done.

1. In the Google Cloud Console I browsed to the Service Accounts module.
2. I clicked on the "App Engine default service account" to see its details.
3. I switched to the "KEYS" tab.
4. I clicked on the "Add Key" button and generated a new JSON key.
5. I downloaded the JSON file and placed it outside of my source code folder.
6. Finally, on my dev machine*, I set the GOOGLE_APPLICATION_CREDENTIALS environment variable to be the path to where I placed the JSON file. More details here: https://cloud.google.com/docs/authentication/production

Once I did this, the call to id_token.fetch_id_token() picked up the service account details from the key file and returned the token successfully.

* - In my case I set the environment variable inside my PyCharm Debug Configuration.