What changes are required when moving simple synapsesql implementation from Spark 2.4.8 to Spark 3.1.2?

Question

I have a simple implementation of.write.synapsesql() method (code shown below) that works in Spark 2.4.8 but not in Spark 3.1.2 ( documentation/example here ). The data in use is a simple notebook-created foobar type table. Searching for key phrases online from and about the error did not turn up any new information for me. What is the cause of the error in 3.1.2?

Spark 2.4.8 version (behaves as desired):

val df = spark.sql("SELECT * FROM TEST_TABLE")
df.write.synapsesql("my_local_db_name.schema_name.test_table", Constants.INTERNAL, None)

Spark 3.1.2 version (extra method is same as in documentation, can also be left out with a similar result):

val df = spark.sql("SELECT * FROM TEST_TABLE")
df.write.synapsesql("my_local_db_name.schema_name.test_table", Constants.INTERNAL, None, 
                     Some(callBackFunctionToReceivePostWriteMetrics))

The resulting error (only in 3.1.2) is:

WriteFailureCause -> java.lang.IllegalArgumentException: Failed to derive `https` scheme based staging location URL for SQL COPY-INTO}

Answer 1

As the documentation from the question states, ensure that you are setting the options correctly with

val writeOptionsWithAADAuth:Map[String, String] = Map(Constants.SERVER -> "<dedicated-pool-sql-server-name>.sql.azuresynapse.net",
                                            Constants.TEMP_FOLDER -> "abfss://<storage_container_name>@<storage_account_name>.dfs.core.windows.net/<some_temp_folder>")

and including the options in your.write statement like so:

df.write.options(writeOptionsWithAADAuth).synapsesql(...)