GCP pipilene between Storage and BigQuery

Question

How can I process the data whenever it's added to the GCP storage bucket? I need to process each file with my own API and add it to BigQuery. I`m looking for a way to trigger the endpoint each time the data is added to the storage bucket.

Answer 1

You can use a function Cloud Storage Trigger, specifically the event type finalize , this event trigger when a write or overwrite a object at your bucket is successfully finalized. When one of these operations is carried out, the function will be executed and within it you could already perform the operations you want, such as sending data to BigQuery.

Here is a tutorial so you can implement and deploy your trigger .