如何在不使用本地服务帐户密钥文件的情况下从 Cloud Run 服务访问 Google Cloud Storage 存储桶？

Question

I just deployed a Cloud Run REST API application, which uses the Google Cloud Storage API to get the last file from a bucket and a folder inside that bucket. This is the code I'm using:

import os
import logging
from flask import Flask
from flask import request, jsonify, render_template
from google.oauth2 import service_account
from google.cloud import storage
from bson.json_util import dumps


app = Flask(__name__)

storage_client = storage.Client.from_service_account_json('sa.json')

@app.route('/')
# API Version 1.0
def index():
    """Welcome to Last File API Version 1.0."""
    button_text = "Add File"
    return render_template('main.html', button_text=button_text)

@app.route("/last_file_m_individual/", methods=["GET"])
def list_m_individual_files():
    """List all files in GCP bucket."""
    bucketName = request.args.get('bucketname')
    bucketFolder = request.args.get('bucketfolder')
    bucket = storage_client.get_bucket(bucketName)
    files = bucket.list_blobs(prefix=bucketFolder)
    fileList = [file.name for file in files if '.' in file.name]
    last_file_pep = fileList[-1]
    last_file_p = last_file_pep.split("/")
    last_file = last_file_p[-1]
    return last_file

@app.errorhandler(500)
def server_error(e):
    # Log the error and stacktrace.
    logging.exception('An error occurred during a request.')
    return 'An internal error occurred.', 500

if __name__ == "__main__":
    app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))

How can I have a service-to-service authentication between Google Cloud Storage and Cloud Run without having locally the Service Account key file for it ?

Thanks in advance.

Answer 1

On Cloud Run, the Cloud Storage client library for Python will automatically pick up the credentials of the identity of the Cloud Run service thanks to the container instance metadata server present inside the container. Read more here

Answer 2

To provide a practical way to achieve this to the great explanation of Steren, you have to replace the line in your code by the following

# Existing line
storage_client = storage.Client.from_service_account_json('sa.json')
# New line
storage_client = storage.Client()

With this line, the service pick the service account provided in the context. Thanks to the metadata server as explained by Steren when you are on Cloud Run. But it's also true on Cloud Function, App Engine and Compute Engine (and similar: Dataflow, Dataproc, GKE,...)

But, how to do this on your local environment? You have 2 solutions

1. You can set the path of your service account key file in the GOOGLE_APPLICATION_CREDENTIALS environment variable.

But, personally, when you develop, I absolutely don't recommend to use service account key file. It's a security weakness. I wrote an article on this and another one to test the containers locally with your own user credential and thus without service account key file .

2. As you could read in the second article mentioned, you can use the ADC (Application Default Credential) in your local environment with the command gcloud auth application-default login .

Thanks to this 2 solutions, you will be able to run your code locally and on Cloud Run (and other GCP services) in the same way without any "if" according with the runtime context.

Answer 3

I just deployed a Cloud Run REST API application, which uses the Google Cloud Storage API to get the last file from a bucket and a folder inside that bucket. This is the code I'm using:

import os
import logging
from flask import Flask
from flask import request, jsonify, render_template
from google.oauth2 import service_account
from google.cloud import storage
from bson.json_util import dumps


app = Flask(__name__)

storage_client = storage.Client.from_service_account_json('sa.json')

@app.route('/')
# API Version 1.0
def index():
    """Welcome to Last File API Version 1.0."""
    button_text = "Add File"
    return render_template('main.html', button_text=button_text)

@app.route("/last_file_m_individual/", methods=["GET"])
def list_m_individual_files():
    """List all files in GCP bucket."""
    bucketName = request.args.get('bucketname')
    bucketFolder = request.args.get('bucketfolder')
    bucket = storage_client.get_bucket(bucketName)
    files = bucket.list_blobs(prefix=bucketFolder)
    fileList = [file.name for file in files if '.' in file.name]
    last_file_pep = fileList[-1]
    last_file_p = last_file_pep.split("/")
    last_file = last_file_p[-1]
    return last_file

@app.errorhandler(500)
def server_error(e):
    # Log the error and stacktrace.
    logging.exception('An error occurred during a request.')
    return 'An internal error occurred.', 500

if __name__ == "__main__":
    app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))

How can I have a service-to-service authentication between Google Cloud Storage and Cloud Run without having locally the Service Account key file for it ?

Thanks in advance.