在Terraform中使用python加载csv数据到bigquery

Question

Read csv file and load it to bigquery through dataflow job - use python coding for this instead of templates How perform this task using terraform(GCP) anyone help

I trying to do it but not understanding what terraform script should I write for it

Answer 1

It's not the responsability of Terraform to deploy a Dataflow job.

There is only a Terraform resource to instantiate a Dataflow template

You can deleguate this to your CI CD.

Example with Beam Python :

• Develop the Job with Beam Python
• Through your CI CD, deploy the Python Beam code to a Cloud Storage bucket
• Run the Dataflow job and main file with Python command line

Example with Beam Java and mvn compile :

• Develop the Job with Beam Java and Maven or Gradle
• Through your CI CD, run mvn compile command to execute the Dataflow job

Example with Beam Java and a fat jar :

• Develop the Job with Beam Java and Maven or Gradle
• Through your CI CD, generate a fat jar
• Deploy this fat jar to a Cloud Storage bucket
• Run the Dataflow job and the Main inside the fat jar with java -jar command

Example with Beam Python and Airflow / Cloud Composer :

• Develop the Job with Beam Python
• Through your CI CD, deploy the Python Beam code to the Cloud Composer bucket with gcloud composer
• In the Airflow code, uses BeamRunPythonPipelineOperator to instantiate the Dataflow job
• Run the Airflow DAG to run the Dataflow job

Example with Beam Java and Airflow / Cloud Composer :

• Develop the Job with Beam Java
• Through your CI CD, generate a fat jar
• Deploy this fat jar to a Cloud Storage bucket
• In the Airflow code, uses BeamRunJavaPipelineOperator to instantiate the Dataflow job targeting on the path of the fat jar
• Run the Airflow DAG to run the Dataflow job