每当在谷歌云存储上创建指定文件时，我想使用云 function 触发 python 脚本

Question

One csv file is uploaded to the cloud storage everyday around 0200 hrs but sometime due to job fail or system crash file upload happens very late. So I want to create a cloud function that can trigger my python bq load script whenever the file is uploaded to the storage.

file_name : seller_data_{date}
bucket name : sale_bucket/

Answer 1

The question lacks enough description of the desired usecase and any issues the OP has faced. However, here are a few possible approaches that you might chose from depending on the usecase.

1. The simple way: Cloud Functions with Storage trigger.

This is probably the simplest and most efficient way of running a Python function whenever a file gets uploaded to your bucket. The most basic tutorial is this .

2. The hard way: App Engine with a few tricks.

Having a basic Flask application hosted on GAE (Standard or Flex), with an endpoint specifically to handle this chek of the files existing, download object, manipulate it and then do something.

This route can act as a custom HTTP triggered function, where once it receives a request (could be from a simple curl request, visit from the browser, PubSub event, or even another Cloud Function).

Once it receives a GET (or POST) request, it downloads the object into the /tmp dir, process it and then do something.

The small benefit with GAE over CF is that you can set a minimum of one instance to stay always alive which means you will not have the cold starts, or risk the request timing out before the job is done.

3. The brutal/overkill way: Clour Run.

Similar approach to App Engine, but with Cloud Run you'll also need to work with the Dockerfile, have in mind that Cloud Run will scale down to zero when there's no usage, and other minor things that apply to building any application on Cloud Run.

########################################

For all the above approaches, some additional things you might want to achieve are the same:

a) Downloading the object and doing some processing on it:

You will have to download it to the /tmp directory as it's the directory for both GAE and CF to store temporary files. Cloud Run is a bit different here but let's not get deep into it as it's an overkill byitself.

However, keep in mind that if your file is large you might cause a high memory usage.

And ALWAYS clean that directory after you have finished with the file. Also when opening a file always use with open... as it will also make sure to not keep files open.

b) Downloading the latest object in the bucket:

This is a bit tricky and it needs some extra custom code. There are many ways to achieve it, but the one I use (always tho paying close attention to memory usage), is upon the creation of the object I upload to the bucket, I get the current time, use Regex to transform it into something like results_22_6 .

What happens now is that once I list the objects from my other script, they are already listed in an accending order. So the last element in the list is the latest object.

So basically what I do then is to check if the filename I have in /tmp is the same as the name of the object[list.length] in the bucket. If yes then do nothing, if no then delete the old one and download the latest one in the bucket.

This might not be optimal, but for me it's kinda preferable.