[英]gcp trigger dataflow job from composer error
我正在尝试使用以下代码从 Composer Airflow DAG 运行数据流作业。
根据代码,我收到两种类型的错误消息。
请建议如何修复它。
a) 错误一:服务账号eMail时被注释(#)
# "serviceAccountEmail": "service-7276363xxxxx@cloudcomposer-accounts.iam.gserviceaccount.com",
错误:
Error: Required 'compute.subnetworks.get' permission for 'projects/vpc-host/regions/us-central1/subnetworks/sbn-dataflow'
b) 错误2:当服务账号eMail被使用时
"serviceAccountEmail": "service-7276363xxxxx@cloudcomposer-accounts.iam.gserviceaccount.com",
错误:
Current user cannot act as service account service-7276363xxxxx@cloudcomposer-accounts.iam.gserviceaccount.com
代码:
import datetime
from airflow import models
from airflow.contrib.operators.dataflow_operator import DataflowTemplateOperator
from airflow.utils.dates import days_ago
bucket_path = models.Variable.get("bucket_path")
project_id = models.Variable.get("project_id")
gce_zone = models.Variable.get("gce_zone")
default_args = {
"owner": "Airflow",
"start_date": days_ago(1),
"depends_on_past": False,
"dataflow_default_options": {
"project": project_id,
"zone": gce_zone,
"serviceAccountEmail": "service-7276363xxxxx@cloudcomposer-accounts.iam.gserviceaccount.com",
"subnetwork": "https://www.googleapis.com/compute/v1/projects/vpc-host/regions/us-central1/subnetworks/sbn-dataflow",
"tempLocation": bucket_path + "/tmp/",
}
}
with models.DAG(
dag_id="composer_dataflow_dag",
default_args=default_args,
schedule_interval=datetime.timedelta(days=1)
) as dag:
dataflow_template_job = DataflowTemplateOperator(
task_id="dataflow_csv_to_bq",
template="gs://dataflow-templates/latest/GCS_Text_to_BigQuery",
parameters={
"javascriptTextTransformFunctionName": "transformCSVtoJSON",
"javascriptTextTransformGcsPath": bucket_path + "/SCORE_STG.js",
"JSONPath": bucket_path + "/SCORE_STG.json",
"inputFilePattern": bucket_path + "/stg_data.csv",
"outputTable": project_id + ":gcp_stage.SCORE_STG",
"bigQueryLoadingTemporaryDirectory": bucket_path + "/tmp/",
},
dag=dag,
)
您必须使用不同的服务帐户。请记住,它必须具有对资源的访问权限。 这应该可以解决这两个问题。
您可以创建一个服务帐户来充当工作人员,如角色分配中所述。 即:一名工人和一名管理员。
除此之外,我没有发现任何不正常的东西。 连参数都正确传递了。 供其他用户参考:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.