[英]Reading first table and execute the inputs in spark SQL and insert into another table
請檢查以下代碼。
創建表 1:
scala> :paste
// Entering paste mode (ctrl-D to finish)
val df = Seq(
(1,"select count(*) from dbnameaaa.tbl_name","select count(*) from dbnameaaa.tbl_name"),
(2,"select count(*) from dbnameaaa.tmp_tbl","select count(*) from dbnameaaa.tmp_tbl"))
.toDF("test_id","execution_script_src","execution_script_target")
// Exiting paste mode, now interpreting.
df: org.apache.spark.sql.DataFrame = [test_id: int, execution_script_src: string ... 1 more field]
scala> df.show(false)
+-------+---------------------------------------+---------------------------------------+
|test_id|execution_script_src |execution_script_target |
+-------+---------------------------------------+---------------------------------------+
|1 |select count(*) from dbnameaaa.tbl_name|select count(*) from dbnameaaa.tbl_name|
|2 |select count(*) from dbnameaaa.tmp_tbl |select count(*) from dbnameaaa.tmp_tbl |
+-------+---------------------------------------+---------------------------------------+
創建查詢執行和條件 UDF
scala> :paste
// Entering paste mode (ctrl-D to finish)
val execute = udf((query: String) => {
try { spark.sql(query).map(_.getAs[Long](0)).collect.head }catch { case _: Exception => 0L }
})
val condition = udf((actual:Long,expected:Long) => {
s"""{"=":"${if (actual == expected) "Pass" else "Fail"}","<":"${if (actual < expected) "Pass" else "Fail"}",">":"${if (actual > expected) "Pass" else "Fail"}","<>":"${if (actual != expected) "Pass" else "Fail"}"}"""
})
// Exiting paste mode, now interpreting.
execute: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,LongType,Some(List(StringType)))
condition: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function2>,StringType,Some(List(LongType, LongType)))
決賽桌結果
scala> :paste
// Entering paste mode (ctrl-D to finish)
df
.withColumn("actual_result",execute($"execution_script_src"))
.withColumn("expected_result",execute($"execution_script_target"))
.withColumn("test_condition",lit("[ =, <, >, <> ]"))
.withColumn("test_result",condition($"actual_result",$"expected_result"))
.withColumn("create_date",current_date)
.withColumn("modify_date",current_date)
.withColumn("created_by",lit(spark.sparkContext.sparkUser))
.withColumn("modified_by",lit(spark.sparkContext.sparkUser))
.withColumn("execute_date",current_date)
.show(false)
// Exiting paste mode, now interpreting.
+-------+---------------------------------------+---------------------------------------+-------------+---------------+---------------+----------------------------------------------+-----------+-----------+----------+-----------+------------+
|test_id|execution_script_src |execution_script_target |actual_result|expected_result|test_condition |test_result |create_date|modify_date|created_by|modified_by|execute_date|
+-------+---------------------------------------+---------------------------------------+-------------+---------------+---------------+----------------------------------------------+-----------+-----------+----------+-----------+------------+
|1 |select count(*) from dbnameaaa.tbl_name|select count(*) from dbnameaaa.tbl_name|11 |11 |[ =, <, >, <> ]|{"=":"Pass","<":"Fail",">":"Fail","<>":"Fail"}|2020-05-06 |2020-05-06 |srinivas |srinivas |2020-05-06 |
|2 |select count(*) from dbnameaaa.tmp_tbl |select count(*) from dbnameaaa.tmp_tbl |11 |22 |[ =, <, >, <> ]|{"=":"Fail","<":"Pass",">":"Fail","<>":"Pass"}|2020-05-06 |2020-05-06 |srinivas |srinivas |2020-05-06 |
+-------+---------------------------------------+---------------------------------------+-------------+---------------+---------------+----------------------------------------------+-----------+-----------+----------+-----------+------------+
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.