Using a service account with boto to access Cloud Storage in GAE with gcs_oauth2_boto_plugin

Question

I am wondering if anyone knows a way to use a service account to authenticate if I want to access data in Cloud Storage by: 1. Using boto library (and gcs_oauth2_boto_plugin) 2. Running in Google App Engine (GAE)

Following https://developers.google.com/storage/docs/gspythonlibrary I am using boto and gcs_oauth2_boto_plugin to authenticate and perform actions against Cloud Storage (upload/download files). I am using a service account to authenticate so that we don't have to authenticate with a Google account periodically (the thought being that if we run this in GCE, it'll run with the GCE service account -- haven't actually done that yet). Locally, I've set up my boto config file to use the service account and point to a p12 key file. This runs fine locally.

I would like to use the same code to interact with Cloud Storage from within Google App Engine (GAE). We are running a light weight ETL process that transforms and loads the data into Big Query. We want to run this code in App Engine task queue (the task is getting triggered by an Object Change Notification from Cloud Storage).

Since we're currently relying on the boto config (~/.boto), I adapted http://thurloat.com/2010/06/07/google-storage-and-app-engine to put the relevant config items for a service account.

When I finally run the code from App Engine (dev_appserver.py), I get the below stack trace:

Traceback (most recent call last):
  File "/home/some-user/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 1536, in __call__
    rv = self.handle_exception(request, response, e)
  File "/home/some-user/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File "/home/some-user/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/home/some-user/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/home/some-user/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/home/some-user/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.1/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/home/some-user/dev/myApp/main.py", line 247, in post
    gs.download(fname, fp)
  File "/home/some-user/dev/myApp/cloudstorage.py", line 107, in download
    bytes = src_uri.get_key().get_contents_to_file(fp)
  File "/home/some-user/dev/myApp/boto/storage_uri.py", line 336, in get_key
    bucket = self.get_bucket(validate, headers)
  File "/home/some-user/dev/myApp/boto/storage_uri.py", line 181, in get_bucket
    conn = self.connect()
  File "/home/some-user/dev/myApp/boto/storage_uri.py", line 140, in connect
    **connection_args)
  File "/home/some-user/dev/myApp/boto/gs/connection.py", line 47, in __init__
    suppress_consec_slashes=suppress_consec_slashes)
  File "/home/some-user/dev/myApp/boto/s3/connection.py", line 190, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "/home/some-user/dev/myApp/boto/connection.py", line 568, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/home/some-user/dev/myApp/boto/auth.py", line 929, in get_auth_handler
    ready_handlers.append(handler(host, config, provider))
  File "/home/some-user/dev/myApp/gcs_oauth2_boto_plugin/oauth2_plugin.py", line 56, in __init__
    cred_type=oauth2_client.CredTypes.OAUTH2_SERVICE_ACCOUNT)
  File "/home/some-user/dev/myApp/gcs_oauth2_boto_plugin/oauth2_helper.py", line 48, in OAuth2ClientFromBotoConfig
    token_cache = oauth2_client.FileSystemTokenCache()
  File "/home/some-user/dev/myApp/gcs_oauth2_boto_plugin/oauth2_client.py", line 175, in __init__
    tempfile.gettempdir(), 'oauth2_client-tokencache.%(uid)s.%(key)s')
  File "/home/some-user/google-cloud-sdk/platform/google_appengine/google/appengine/dist/tempfile.py", line 61, in PlaceHolder
    raise NotImplementedError("Only tempfile.TemporaryFile is available for use")
NotImplementedError: Only tempfile.TemporaryFile is available for use

Looks like the problem is just with gcs_oauth2_boto_plugin trying to use a temporary directory when caching the oauth credentials (App Engine only supports tempfile.TemporaryFile).

Rather than try and patch gcs_oauth2_boto_plugin, is there potentially another solution? Can we use a service account with gcs_oauth2_boto_plugin/boto on App Engine to access Cloud Storage resources?

Or, am I using the wrong authentication method here?

Answer 1

This doesn't quite answer the question directly, but instead of using boto and gcs_oauth2_boto_plugin, I am using the "Google Cloud Storage Python Client Library", GoogleAppEngineCloudStorageClient from pip.

https://developers.google.com/appengine/docs/python/googlecloudstorageclient/