[英]What changes are required when moving simple synapsesql implementation from Spark 2.4.8 to Spark 3.1.2?
I have a simple implementation of.write.synapsesql() method (code shown below) that works in Spark 2.4.8 but not in Spark 3.1.2 ( documentation/example here ).我有一个简单的 .write.synapsesql() 方法实现(代码如下所示),它在 Spark 2.4.8 中有效,但在 Spark 3.1.2 中无效( 此处为文档/示例)。 The data in use is a simple notebook-created foobar type table.
使用的数据是一个简单的笔记本创建的 foobar 类型表。 Searching for key phrases online from and about the error did not turn up any new information for me.
在网上搜索有关错误的关键短语并没有为我提供任何新信息。 What is the cause of the error in 3.1.2?
3.1.2错误的原因是什么?
Spark 2.4.8 version (behaves as desired): Spark 2.4.8 版本(按需要运行):
val df = spark.sql("SELECT * FROM TEST_TABLE")
df.write.synapsesql("my_local_db_name.schema_name.test_table", Constants.INTERNAL, None)
Spark 3.1.2 version (extra method is same as in documentation, can also be left out with a similar result): Spark 3.1.2版本(额外方法同文档,也可以省略,结果类似):
val df = spark.sql("SELECT * FROM TEST_TABLE")
df.write.synapsesql("my_local_db_name.schema_name.test_table", Constants.INTERNAL, None,
Some(callBackFunctionToReceivePostWriteMetrics))
The resulting error (only in 3.1.2) is:由此产生的错误(仅在 3.1.2 中)是:
WriteFailureCause -> java.lang.IllegalArgumentException: Failed to derive `https` scheme based staging location URL for SQL COPY-INTO}
As the documentation from the question states, ensure that you are setting the options correctly with正如问题中的文档所述,请确保您正确设置了选项
val writeOptionsWithAADAuth:Map[String, String] = Map(Constants.SERVER -> "<dedicated-pool-sql-server-name>.sql.azuresynapse.net",
Constants.TEMP_FOLDER -> "abfss://<storage_container_name>@<storage_account_name>.dfs.core.windows.net/<some_temp_folder>")
and including the options in your.write statement like so:并在 your.write 语句中包括选项,如下所示:
df.write.options(writeOptionsWithAADAuth).synapsesql(...)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.