Write data using JDBC connection to Azure SQL DB with Scala code Databricks notebook

Question

I am trying to insert data from hive table to Azure SQL DB table. The SQL DB table already exists and I just want to overwrite data into it with following Scala JDBC write code. This code is writing data to SQL DB table, however it is changing its DDL (datatypes/column names). How can I avoid it. I want simple insert on table.

Answer 1

You can see this document: Connecting to SQL Databases using JDBC . It provides you some examples about write data to JDBC .

This section shows how to write data to a database from an existing Spark SQL table named diamonds.

%sql -- quick test that this test table exists
select * from diamonds limit 5

The following code saves the data into a database table named diamonds. Using column names that are reserved keywords can trigger an exception. The example table has column named table, so you can rename it with withColumnRenamed() prior to pushing it to the JDBC API.

spark.table("diamonds").withColumnRenamed("table", "table_number")
     .write
     .jdbc(jdbcUrl, "diamonds", connectionProperties)

Spark automatically creates a database table with the appropriate schema determined from the DataFrame schema.

The default behavior is to create a new table and to throw an error message if a table with the same name already exists. You can use the Spark SQL SaveMode feature to change this behavior. For example, here's how to append more rows to the table:

import org.apache.spark.sql.SaveMode

spark.sql("select * from diamonds limit 10").withColumnRenamed("table", "table_number")
     .write
     .mode(SaveMode.Append) // <--- Append to the existing table
     .jdbc(jdbcUrl, "diamonds", connectionProperties)

You can also overwrite an existing table:

spark.table("diamonds").withColumnRenamed("table", "table_number")
     .write
     .mode(SaveMode.Overwrite) // <--- Overwrite the existing table
     .jdbc(jdbcUrl, "diamonds", connectionProperties)

Hope this helps.