SQL 數據庫使用 JDBC + 參數化 SQL 查詢 + Databricks

Question

在 Databricks 中，我將 SQL 表讀取為

val TransformationRules = spark.read.jdbc(jdbcUrl, "ADF.TransformationRules", connectionProperties)
.select("RuleCode","SourceSystem","PrimaryTable", "PrimaryColumn", "SecondaryColumn", "NewColumnName","CurrentFlag")
.where("SourceSystem = 'QWDS' AND RuleCode = 'STD00003' ")

如何在Where子句中參數化SourceSystem和RuleCode

指的是： https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/sql-databases

Answer 1

如果您導入 spark 隱式，您可以使用美元$插值器創建對列的引用。 此外，您可以使用帶有列的 API 來制作邏輯，它會是這樣的。

val sourceSystem = "QWDS"
val ruleCode = "STD00003"

import spark.implicits._
val TransformationRules = spark.read.jdbc(jdbcUrl, "ADF.TransformationRules", connectionProperties)
.select("RuleCode","SourceSystem","PrimaryTable", "PrimaryColumn", "SecondaryColumn", "NewColumnName","CurrentFlag")
.where($"SourceSystem" === sourceSystem && $"RuleCode" === ruleCode)

val ssColumn: Column = $"SourceSystem"

As you can see, the dollar will provide a Column object, with logic like cooperation, casting renaming etc. In combination with the functions in org.apache.spark.sql.function will allow you to implement almost all you need.

Answer 2

據我正確理解您的問題，您想將值插入到 where 子句字符串中嗎？ 也許下面的解決方案可能對您有用：

val TransformationRules = spark.read.jdbc(jdbcUrl, "ADF.TransformationRules", connectionProperties)
.select("RuleCode","SourceSystem","PrimaryTable", "PrimaryColumn", "SecondaryColumn", "NewColumnName","CurrentFlag")
.where("SourceSystem = '{}' AND RuleCode = '{}' ".format(sourceSystem, ruleCode))