write to a JDBC source in scala

Question

I am trying to write classic sql query using scala to insert some information into a sql server database table.

The connection to my database works perfectly and I succeed to read data from JDBC, from a table recently created called "textspark" which has only 1 column called "firstname" create table textspark(firstname varchar(10)) .

However, when I try to write data into the table , I get the following error:

Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: textspark

this is my code:

//Step 1: Check that the JDBC driver is available
Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver")

//Step 2: Create the JDBC URL
val jdbcHostname = "localhost"
val jdbcPort = 1433
val jdbcDatabase ="mydatabase"
val jdbcUsername = "mylogin"
val jdbcPassword = "mypwd"

// Create the JDBC URL without passing in the user and password parameters.
val jdbcUrl = s"jdbc:sqlserver://${jdbcHostname}:${jdbcPort};database=${jdbcDatabase}"

// Create a Properties() object to hold the parameters.
import java.util.Properties
val connectionProperties = new Properties()

connectionProperties.put("user", s"${jdbcUsername}")
connectionProperties.put("password", s"${jdbcPassword}")

//Step 3: Check connectivity to the SQLServer database
val driverClass = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
connectionProperties.setProperty("Driver", driverClass)



//Read data from JDBC
val textspark_table = spark.read.jdbc(jdbcUrl, "textspark", connectionProperties)
textspark_table.show()

//the read operation works perfectly!!


//Write data to JDBC
import org.apache.spark.sql.SaveMode


spark.sql("insert into textspark values('test') ")
     .write
     .mode(SaveMode.Append) // <--- Append to the existing table
     .jdbc(jdbcUrl, "textspark", connectionProperties)

//the write operation generates error!!

Can anyone help me please to fix this error?

Answer 1

You don't use insert statement in Spark . You specified the append mode what is ok. You shouldn't insert data, you should select / create it. Try something like this:

spark.sql("select 'text'")
  .write
  .mode(SaveMode.Append)
  .jdbc(jdbcUrl, "textspark", connectionProperties)

or

Seq("test").toDS
  .write
  .mode(SaveMode.Append)
  .jdbc(jdbcUrl, "textspark", connectionProperties)