如何最好地使用云功能处理来自谷歌长期运行操作的异步响应

Question

I'm using Google Cloud Functions (python) to initiate an asset inventory export from GCP by calling the exportAssets() method here . The method returns an Operations object defined here which can be used to poll the operation until it is complete. Of course since this is a cloud function I'm limited to 540 seconds so cannot do that forever. The google api python client offers the add_done_callback() method where one can await an async response, but as far as I can tell it requires me to keep a thread alive within the cloud function. Is there a way to tell the Asset Inventory API executing the operation to send the aync response (success or failure) to a pubsub topic where I can properly handle the response? Trying to avoid spinning up an appengine instance with basic_scaling to support 24 hour timeouts.

    from google.cloud import asset_v1
    # .....
    # Setup request to asset inventory API
    parent = "organizations/{}".format(GCP_ORGANIZATION)
    requested_type = 'RESOURCE'
    dataset = 'projects/{}/datasets/gcp_assets_{}'.format(GCP_PROJECT, requested_type)
    partition_spec = asset_v1.PartitionSpec
    partition_key = asset_v1.PartitionSpec.PartitionKey.REQUEST_TIME
    partition_spec.partition_key = asset_v1.PartitionSpec.PartitionKey.REQUEST_TIME

    output_config = asset_v1.OutputConfig()
    output_config.bigquery_destination.dataset = dataset
    output_config.bigquery_destination.table = 'assets'
    output_config.bigquery_destination.separate_tables_per_asset_type = True
    output_config.bigquery_destination.partition_spec.partition_key = partition_key

    # Make API request to asset inventory API
    print("Creating job to load 'asset types: {}' to {}".format(
        requested_type,
        dataset
    ))
    response = ASSET_CLIENT.export_assets(
        request={
            "parent": parent,
            "content_type": content_type,
            "output_config": output_config,
        }
    )
    print(response.result())  # This waits for the job to complete

Answer 1

Cloud Asset inventory export doesn't offer a PubSub notification at the end of the export. However, in my previous company, it took about 5 minutes to export 100k+ assets; it's not so bad, And if you have more assets. I'm sure you can contact Google Cloud (use your Customer Engineer) to add this notification in the roadmap.

---

Anyway, if you want to build a workaround, you can use workflows .

• Use a Cloud Function to trigger your workflow
• In your workflow,
  • Call the Cloud Asset API to export data to BigQuery
  • Get the response and perform a loop (test the export job status, if not OK, sleep X seconds and test again)
  • When the job is over, call PubSub API (or directly a Cloud Function) to submit the job status and process it.