自动化Python脚本的最佳方法

Question

I've got a number of scripts that move data from various data sources to BigQuery. Those scripts need to be run daily. I can't run it on my machine as it may be offline at the time when the script is to be run.

What service should i use to automate this and what should i look for when choosing one? Is there anything like this within Google / Google Cloud environment?

And how the process of automating the script once i have the server will look like? Appreciate any advice or links to the docs.

Answer 1

You have two options:

1. Spin up a GCE (or EC2) instance, and use a cron job to kick off your python scripts.
2. Forget about cron tabs, and use Apache Airflow (see links below).

The latter would be my preferred weapon of choice.

https://airflow.incubator.apache.org/

http://engineering.pmc.com/2017/03/playing-around-with-apache-airflow-bigquery-62/

https://medium.com/google-cloud/airflow-for-google-cloud-part-1-d7da9a048aa4

Answer 2

Reliable Task Scheduling on Google Compute Engine you can use this and set daily execution time for script.

Boto allows you to write some scripts to automate things like starting AWS EC2 instances Boto

Answer 3

Jenkins is a useful tool for this sort of thing. It's meant for CI/CD but can be used to automate jobs at certain times or after certain conditions have been met.

Once you install it you access it through your browser at localhost:8080 and set up your build conditions.

There's about a million plug-ins you can easily browse within the GUI so you should find something there to help.