Is it possible for BigQuery to call an endpoint when it has finished uploading a file from Cloud Storage to BQ or when inserting data via Streaming?

Question

I am uploading files to BigQuery from Cloud Storage and via streaming and I would like to know if when BigQuery finishes loading the data in a table it can call an endpoint of my choice.

For example, a file from Storage:

load_job = bq_client.load_table_from_uri(
                source_uris=uri,
                destination=destination_table,
                job_config=job_config, endpoint="http://my-endpoint.com"
            )

Or for example, in streaming:

rows_to_insert = [
    { "name": "John", "age": 22},
    {"name": "Will", "age": 33},
]

errors = bq_client.insert_rows_json(table_id, rows_to_insert, endpoint="http://my-endpoint.com)

In my endpoint I have a python code that I want to execute when the data has been inserted in BigQuery.

The code that inserts the data (streaming or from Storage) should not wait for BigQuery to confirm that the data has been inserted, this should be done by BigQuery calling the endpoint.

Is there a way to do this?

Answer 1

For that use case, you can use Eventarc . Eventarc captures events based on the audit logs and generate an event to call Cloud Run and Cloud Functions V2 (Cloud Workflows also, and many more endpoints are coming).

You have to check in the event, if the status in done or not. You should receive an event for each state update.

---

The other solution is to trigger your load job from Cloud Workflow and to use the blocking BigQuery job insert connector that wait the end of the load job before going to the next step