[英]How to run a Python script every day in GCP?
I have a simple script, it retrieves data from an API and loads it into BigQuery.我有一个简单的脚本,它从 API 检索数据并将其加载到 BigQuery 中。 I was using Cloud Functions and it was running smoothly, however there came a time that reached the 9-minute run time limit.
我使用的是 Cloud Functions,它运行平稳,但是有一次达到了 9 分钟的运行时间限制。
What is the best way to do this in GCP, taking the time the script needs?花脚本需要的时间在 GCP 中执行此操作的最佳方法是什么? I was thinking of creating another Cloud Function that daily starts a preemtible VM, the VM executes the script and in the end turns itself off.
我正在考虑创建另一个 Cloud Function,它每天启动一个抢占式 VM,VM 执行脚本并最终自行关闭。 To keep the price low, the VM would always shut down at the end of the data load.
为了保持低价,VM 总是在数据加载结束时关闭。 It would start the next day at the selected time.
它将在第二天的选定时间开始。
I don't know where to start to do this, but I was wondering if that would be the best way.我不知道从哪里开始这样做,但我想知道这是否是最好的方法。
Cloud functions aren't really suited to batch jobs that may be longer running than 10 minutes.云功能并不真正适合运行时间可能超过 10 分钟的批处理作业。 I'd suggest running your job using a Compute Engine VM and scheduling it with a combination of Cloud functions / Cloud scheduler.
我建议使用 Compute Engine 虚拟机运行您的工作,并结合云功能/云调度程序来调度它。
Here's a rough outline:这是一个粗略的大纲:
import googleapiclient.discovery
def start_job(event, context):
"""Triggered from a message on a Cloud Pub/Sub topic.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
compute = googleapiclient.discovery.build('compute', 'v1')
compute.instances().insert(
project='project_id',
zone='us-east1-b',
body=vm_config).execute()
This lets you avoid the cost of an always-on VM.这使您可以避免永远在线 VM 的成本。 See this blog post for more detail.
有关更多详细信息,请参阅此博客文章。
Could this work?这能行吗?
import schedule
import time
def run_daily():
do something
do something else
schedule.every().day.at("08:20:30").do(run_daily) # HH MM SS
while True:
schedule.run_pending()
time.sleep(1)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.