如何通过本地 Docker 容器中的 python 应用程序从云存储桶中读取文件

Question

Let me preface this with the fact that I am fairly new to Docker, Jenkins, GCP/Cloud Storage and Python.

Basically, I would like to write a Python app, that runs locally in a Docker container (alpine3.7 image) and reads chunks, line by line, from a very large text file that is dropped into a GCP cloud storage bucket. Each line should just be output to the console for now.

I learn best by looking at working code, I am spinning my wheels trying to put all the pieces together using these technologies (new to me).

I already have the key file for that cloud storage bucket on my local machine.

I am also aware of these posts:

• How to Read .json file in python code from google cloud storage bucket .
• Lazy Method for Reading Big File in Python?

I just need some help putting all these pieces together into a working app.

I understand that I need to set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of the key file in the container. However, I don't know how to do that in a way that works well for multiple developers and multiple environments (Local, Dev, Stage and Prod).

Answer 1

This is just a simple quickstart (I am sure it can be done better) to read a file from a Google Cloud Storage bucket via a python app (Docker container deployed to Google Cloud Run):

You can find more information here link

1. Create a directory with the following files:

   a. app.py

   import os from flask import Flask from google.cloud import storage app = Flask(__name__) @app.route('/') def hello_world(): storage_client = storage.Client() file_data = 'file_data' bucket_name = 'bucket' temp_file_name = 'temp_file_name' bucket = storage_client.get_bucket(bucket_name) blob = bucket.get_blob(file_data) blob.download_to_filename(temp_file_name) temp_str='' with open (temp_file_name, "r") as myfile: temp_str = myfile.read().replace('\\n', '') return temp_str if __name__ == "__main__": app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))

   b.Dockerfile

   # Use an official Python runtime as a parent image FROM python:2.7-slim # Set the working directory fo /app WORKDIR /app # Copy the current directory contents into the container /app COPY . /app # Install any needed packages specified in requirements.txt RUN pip install --trusted-host pypi.python.org -r requirements.txt RUN pip install google-cloud-storage # Make port 80 available to the world outside the container EXPOSE 80 # Define environment variable ENV NAME World # Run app.py when the container launches CMD ["python", "app.py"]

   c. requirements.txt

    Flask==1.1.1 gunicorn==19.9.0 google-cloud-storage==1.19.1

2. Create a service account to access the storage form Cloud Run:

    gcloud iam service-accounts create cloudrun --description 'cloudrun'

3. Set the permission of the service account:

    gcloud projects add-iam-policy-binding wave25-vladoi --member serviceAccount:cloud-run@project.iam.gserviceaccount.com --role roles/storage.admin

4. Build the container image:

    gcloud builds submit --tag gcr.io/project/hello

5. Deploy the application to Cloud Run:

    gcloud run deploy --image gcr.io/project/hello --platform managed ----service-account cloud-run@project.iam.gserviceaccount.com

EDIT :

One way to develop locally is :

1. Your Dev Opp Team will get the service account key.json:

    gcloud iam service-accounts keys create ~/key.json --iam-account serviceAccount:cloudrun@project.iam.gserviceaccount.com

2. Store the key.json file in the same working directory

3. The Dockerfile command `COPY . /app ' will copy the file to Docker container

4. Change the app.py to :

    storage.Client.from_service_account_json('key.json')