使用 Google Cloud Data Fusion 执行自定义 SQL 查询

Question

I have data pipelines that consist of multiple SQL queries being run against BigQuery tables, I would like to build these in Google Cloud Fusion, but I don't see an option to transform/select with custom SQL.

is this available, or am I misinterpreting the use cases for this tool?

Answer 1

A new Action plugin is being added that would allow you to specify a SQL to run in BQ. Expect the connectors to be available in Hub by mid May.

Nitin

Answer 2

There is now a native BigQuery Execute action that allows SQL queries to run as part of a Data Fusion Pipeline.

This job is an action, see below from the official documentation :

Action plugins define custom actions that are scheduled to take place during a workflow but don't directly manipulate data in the workflow. For example, using the Database custom action, you can run an arbitrary database command at the end of your pipeline. Alternatively, you can trigger an action to move files within Cloud Storage.