[英]How to create multiple glue jobs as one glue job in terraform
我是 terraform 腳本的新手,我想創建多個粘合作業,每個作業包含不同的名稱和不同的腳本。 有沒有可能在變量的幫助下將這多個工作創建為一個工作?
例如:變量.tf
variable "db2jobnames" {
description = "db2 glue job names"
type = list
default = ["sql_db_job", "sql_db_job2"]
}
variable "script_location" {
description = "db2 glue job scripts"
type = list
default = ["s3://s3_buget/sql_db_job.py", "s3://s3_buget/sql_db_job.py"]
}
膠水連接.tf
resource "aws_glue_connection" "conn_db2" {
count = var.created_CR ? 1 : 0
connection_type = "JDBC"
connection_properties = {
JDBC_CONNECTION_URL = "jdbc:db2://lkidjhyft:50000/ZXHAG006G"
PASSWORD = "acfg3"
USERNAME = "ndhygsf"
}
name = "${var.department}-${var.application}-connection"
physical_connection_requirements {
availability_zone = var.connection_availability_zone
security_group_id_list = data.aws_security_groups.AWS_Public_Services.ids
subnet_id = data.aws_subnet.selected.id
}
}
和我的膠水工作。 主程序
resource "aws_glue_job" "etl_jobs" {
count = var.created_GL ? 1 : 0
count = "${length(var.db2jobnames)}"
count = "${length(var.script_location)}"
name = "${var.db2jobnames[count.index]}_db2etljobs"
role_arn = aws_iam_role.glue_role.arn
command {
python_version = var.python_version
script_location = "${var.script_location[count.index]}"
}
default_arguments = {
"--extra-jars" = "${var.JarDir}"
"--TempDir" = "${var.TempDir}"
"--class" = "GlueApp"
"--enable-continuous-cloudwatch-log" = "${var.enable-continuous-cloudwatch_log}"
"--enable-glue-datacatalog" = "${var.enable-glue-datacatalog}"
"--enable-metrics" = "${var.enable-metrics}"
"--enable-spark-ui" = "${var.enable-spark-ui}"
"--job-bookmark-option" = "${var.job-bookmark-option}"
"--job-language" = "python"
"--env" = "${var.paramregion}"
"--spark-event-logs-path" = "${var.sparkeventlogpath}"
}
execution_property {
max_concurrent_runs = var.max_concurrent_runs
}
connections = [
"${aws_glue_connection.conn_db2[count.index].name}"
]
glue_version = var.glue_version
max_retries = 0
worker_type = var.worker_type
number_of_workers = 20
timeout = 2880
tags = local.common_tags
}
我試圖插入兩個計數,但出現錯誤。 我們怎么能用一份工作創造兩個工作呢? 就像一個作業需要創建第一個數據庫名稱和第一個腳本位置,如下所示。
job1--> sql_db_job - s3://s3_buget/sql_db_job.py
job2--> sql_db_job2 - s3://s3_buget/sql_db_job2.py
任何答復將不勝感激。 謝謝你。
根據您提供的變量和代碼,您必須更改count
,以便它使用其中一個列表的長度。 例如:
resource "aws_glue_job" "etl_jobs" {
count = var.created_GL ? length(var.db2jobnames) : 0
name = "${var.db2jobnames[count.index]}_db2etljobs"
role_arn = aws_iam_role.glue_role.arn
command {
python_version = var.python_version
script_location = var.script_location[count.index]
}
default_arguments = {
"--extra-jars" = var.JarDir
"--TempDir" = var.TempDir
"--class" = "GlueApp"
"--enable-continuous-cloudwatch-log" = var.enable-continuous-cloudwatch_log
"--enable-glue-datacatalog" = var.enable-glue-datacatalog
"--enable-metrics" = var.enable-metrics
"--enable-spark-ui" = var.enable-spark-ui
"--job-bookmark-option" = var.job-bookmark-option
"--job-language" = "python"
"--env" = var.paramregion
"--spark-event-logs-path" = var.sparkeventlogpath
}
execution_property {
max_concurrent_runs = var.max_concurrent_runs
}
connections = [
aws_glue_connection.conn_db2[0].name
]
glue_version = var.glue_version
max_retries = 0
worker_type = var.worker_type
number_of_workers = 20
timeout = 2880
tags = local.common_tags
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.