[英]Implementing a cloud function to publish to pubsub triggered by GCS finalize
I've been trying to write and deploy a cloud function in Python. 我一直在尝试用Python编写和部署云功能。 (gave up on node.js due to the messy documentation and relatively fast pace of changes) (由于文档混乱且更改速度相对较快,因此放弃了node.js)
It is meant to publish a message to a Pub/Sub topic, triggered when a file finishes being uploaded into google cloud bucket ("finalize"). 它旨在将消息发布到发布/订阅主题,当文件完成上传到Google云存储桶(“完成”)时触发。
The code I use to deploy the function is 我用来部署该功能的代码是
gcloud functions deploy hello_gcs_generic --runtime python37 --trigger-resource bucketcfpubsub
I have been trying by using this script provided by Google 我一直在尝试使用Google提供的此脚本
import time
from google.cloud import pubsub_v1
project_id = "bucketcfpubsub"
topic_name = "projects/bucketcfpubsub/topics/pubsub"
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_name)
def callback(message_future):
# When timeout is unspecified, the exception method waits indefinitely.
if message_future.exception(timeout=30):
print('Publishing message on {} threw an Exception {}.'.format(
topic_name, message_future.exception()))
else:
print(message_future.result())
for n in range(1, 10):
data = u'Message number {}'.format(n)
# Data must be a bytestring
data = data.encode('utf-8')
# When you publish a message, the client returns a Future.
message_future = publisher.publish(topic_path, data=data)
message_future.add_done_callback(callback)
print('Published message IDs:')
# We must keep the main thread from exiting to allow it to process
# messages in the background.
while True:
time.sleep(60)
To which I receive these errors in Google Cloud Console 我在Google Cloud Console中收到这些错误
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function load error: Code in file main.py can't be loaded.
Detailed stack trace: Traceback (most recent call last):
File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 256, in check_or_load_user_function
_function_handler.load_user_function()
File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 166, in load_user_function
spec.loader.exec_module(main)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/user_code/main.py", line 3, in <module>
from google.cloud import pubsub_v1
ImportError: cannot import name 'pubsub_v1' from 'google.cloud' (unknown location)
Following instruction from these two posts, I've copied the requirements.txt from the helloworld code sample, containing just 按照这 两篇文章的说明,我从helloworld代码示例中复制了requirements.txt,其中仅包含
google-cloud-error-reporting==0.30.0
and updated other cloud functions like bigquery, storage, and logging. 并更新了其他云功能,例如bigquery,存储和日志记录。 I then got these errors: 然后我得到了这些错误:
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function load error: Code in file main.py can't be loaded.
Detailed stack trace: Traceback (most recent call last):
File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 256, in check_or_load_user_function
_function_handler.load_user_function()
File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 166, in load_user_function
spec.loader.exec_module(main)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/user_code/main.py", line 3, in <module>
from google.cloud import pubsub_v1`
and I also found [this thread]( ImportError: cannot import name 'pubsub_v1' from 'google.cloud' (unknown location) but I don't really understand what's the solution, I've tried replacing pubsub_v1 with google-cloud-pubsub==0.38.0 which did not help. I get this error instead: 并且我还发现了[this thread]( ImportError:无法从'google.cloud'(未知位置)导入名称'pubsub_v1',但我不太了解解决方案,我尝试用google-cloud-pubsub替换pubsub_v1 == 0.38.0并没有帮助,而是出现此错误:
Deploying function (may take a while - up to 2 minutes)...failed.
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function load error: Code in file main.py can't be loaded.
Detailed stack trace: Traceback (most recent call last):
File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 256, in check_or_load_user_function
_function_handler.load_user_function()
File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 166, in load_user_function
spec.loader.exec_module(main)
File "<frozen importlib._bootstrap_external>", line 724, in exec_module
File "<frozen importlib._bootstrap_external>", line 860, in get_code
File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/user_code/main.py", line 3
Also, this does not seem to be a sustainable fix if the code will break once Google updates pubsub to a new version? 另外,如果Google将pubsub更新到新版本后代码会中断,这似乎不是可持续的解决方案?
So I'm very much a beginner and quite lost, but I hope this documentation could help you guys help me. 因此,我非常初学者,很迷失,但我希望本文档可以对您有所帮助。
UPDATE: 更新:
It seems pubsub and pubsub_v1 are both valid to be used, not sure what's the difference, though. 似乎pubsub和pubsub_v1都可以使用,尽管不确定有什么区别。
@dustin I did a pip install -r requirements.txt which ended up matchiing what you provided. @dustin我做了一个pip install -r requirements.txt,最终匹配了您提供的内容。 I've also noticed an error in deploying the function as 'hello-gcs-generic', which should be changed to 'callback'. 我还注意到在将函数部署为“ hello-gcs-generic”时出错,应将其更改为“ callback”。
The python code now runs well locally, but deploying it to the cloud using the above code (the first code line in the OP) consistently returns this error python代码现在在本地运行良好,但是使用上述代码(OP中的第一行代码)将其部署到云中始终会返回此错误
ERROR: (gcloud.functions.deploy) OperationError: code=3, messa
ge=Function load error: Error: function load attempt timed out
.
You need to add google-cloud-pubsub
to your requirements.txt
file, not in your main.py
file. 您需要将google-cloud-pubsub
添加到您的requirements.txt
文件中,而不是在main.py
文件中。 It should look like this: 它看起来应该像这样:
google-cloud-error-reporting==0.30.0
google-cloud-pubsub==0.38.0
There is a simpler Python quickstart example that does what you need. 有一个更简单的Python快速入门示例可以满足您的需求。 ;-) ;-)
The example you are quoting is more advanced. 您引用的示例更为高级。 It shows how to publish a message with error-handling. 它显示了如何发布带有错误处理的消息。 The while(True): sleep(60)
line in the advanced example is there to keep the main thread alive, unless Ctrl+C
or its equivalent is issued to stop the program from running. 在高级示例中, while(True): sleep(60)
行可以使主线程保持活动状态,除非发出Ctrl+C
或等效的命令来停止程序运行。 The reason for this sleep
function to be there is so we can wait for the callback calls on the publish futures to finish and not exit the program immediately after the publish call. 该sleep
函数存在的原因是,因此我们可以等待发布期货上的回调调用完成,而不必在发布调用后立即退出程序。 Again, this is perhaps a little too complex for what you are trying to learn to do with Cloud Pub/Sub with Cloud Functions. 再次,对于您要学习如何使用具有Cloud Functions的Cloud Pub / Sub而言,这可能有点太复杂了。 I would suggest to steer away from the advanced example and use the quickstart example. 我建议不要使用高级示例,而使用快速入门示例。
from google.cloud import pubsub_v1
# TODO project_id = "Your Google Cloud Project ID"
# TODO topic_name = "Your Pub/Sub topic name"
publisher = pubsub_v1.PublisherClient()
# The `topic_path` method creates a fully qualified identifier
# in the form `projects/{project_id}/topics/{topic_name}`
topic_path = publisher.topic_path(project_id, topic_name)
for n in range(1, 10):
data = u'Message number {}'.format(n)
# Data must be a bytestring
data = data.encode('utf-8')
# When you publish a message, the client returns a future.
future = publisher.publish(topic_path, data=data)
print('Published {} of message ID {}.'.format(data, future.result()))
print('Published messages.')
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.