How to signed url for Google cloud storage on Google App Engine Standard environment with Python3.7?

Question

I can't find a way to have a working signed url on Google App Engine Standard environment with Python3.7.

I have look at the documentation here : https://cloud.google.com/storage/docs/access-control/signing-urls-manually

Within a Google App Engine application, you can use the App Engine App Identity service to sign your string.

But the App Engine App Identity rely on google.appengine package, that is not availalble on python 3.7 env as explain here

Proprietary App Engine APIs are not available in Python 3. This section lists recommended replacements.

The overall goal is that your app should be fully portable and run in any standard Python environment. You write a standard Python app, not an App Engine Python app. As part of this shift, you are no longer required to use proprietary App Engine APIs and services for your app's core functionality. At this time, App Engine APIs are not available in the Python 3.7 runtime.

All the api on sdk rely on google.appengine and raise an exception on python 3.7 env : EnvironmentError('The App Engine APIs are not available.') raise here that rely on proprietary api :

try:
    from google.appengine.api import app_identity
except ImportError:
    app_identity = None

I know I can use many solution like ServiceAccountCredentials.from_json_keyfile_dict(service_account_dict) but I have to upload a file with credentials directly on app engine and I can't do it since the project credential will be expose on git or ci.

I really want to rely on default credential from app engine like other Google Cloud api like storage.Client() for example that work out of box.

Any suggestion ?

Answer 1

For Python interactions with Google Cloud use Python Client that is supported on App Engine standard Python 3 runtime.

To access Cloud Storage using google-cloud-storage from App Engine Standard:

1. Add dependency to the requirements.txt > google-cloud-storage==1.14.0
2. Use Storage Client library , authenticating with storage.Client() only.

Depending on what you need to achieve, I would also suggest trying different possible approaches:

1. Allow anonymous access for public data stored in the bucket.
2. For signed URL API call use Method: projects.serviceAccounts.signBlob . Documentation includes examples:
   • It is important to grant correct permissions to create tokens for Service account
   • You can also check how to use the API - explained on SO .
   • This example explains how to implement signing of the bucket URL using python

It is also possible to sign blobs with appengine api using:

google.appengine.api.app_identity.sign_blob()

Answer 2

This question might be old, but it's one the first ones to show on a Google search, so I thought it might help someone who comes looking in the future to post this here as well.

The answer @guillaume-blaquiere posted here does work, but it requires an additional step not mentioned, which is to add the Service Account Token Creator role in IAM to your default service account , which will allow said default service account to "Impersonate service accounts (create OAuth2 access tokens, sign blobs or JWTs, etc)."

This allows the default service account to sign blobs, as per the signBlob documentation .

I tried it on AppEngine and it worked perfectly once that permission was given.

import datetime as dt

from google import auth
from google.cloud import storage

# SCOPES = [
#     "https://www.googleapis.com/auth/devstorage.read_only",
#     "https://www.googleapis.com/auth/iam"
# ]

credentials, project = auth.default(
#     scopes=SCOPES
)
credentials.refresh(auth.transport.requests.Request())

expiration_timedelta = dt.timedelta(days=1)

storage_client = storage.Client(credentials=credentials)
bucket = storage_client.get_bucket("bucket_name")
blob = bucket.get_blob("blob_name")

signed_url = blob.generate_signed_url(
    expiration=expiration_timedelta,
    service_account_email=credentials.service_account_email,
    access_token=credentials.token,
)

I downloaded a key for the AppEngine default service account to test locally, and in order to make it work properly outside of the AppEngine environment, I had to add the proper scopes to the credentials, as per the commented lines setting the SCOPES . You can ignore them if running only in AppEngine itself.