Apache Spark-JDBC源

Question

Did anyone managed to pull data out or at least connect to an RDBMS through JDBC with their new feature released in 1.3 using their built-in source for Spark SQL instead of RDDJdbc?

https://databricks.com/blog/2015/03/24/spark-sql-graduates-from-alpha-in-spark-1-3.html

I've tried to apply the example mentioned in the post above but that didn't work as it gives me an error. Thought maybe someone can provide me a full example in scala of how to connect and query the data.

Answer 1

Yes. There are two ways of doing it.

1. Programmatically using SQLContext load function.

The load function loads the JDBC datasource data as a DataFrame. If you would like to make this DataFrame available as a table in subsequent Spark SQL queries, it has to be registered using

yourDataFrame.registerTempTable("yourTableName")

If you'd like a complete example, check my blog post .

2. Using SQL

I haven't tried this yet. Based on what I read from documentation, this can be done like the example below.

CREATE TEMPORARY TABLE yourTableName
USING org.apache.spark.sql.jdbc
OPTIONS (
  url "jdbc:postgresql:dbserver",
  dbtable "schema.tablename"
)

As you can guess, it does both loading data as well as registering it as a table in the same command.

I was thinking that SQLContext.sql function can be used to execute the above SQL DDL statement. But it throws the same error you mentioned.

failure: ''insert'' expected but identifier CREATE found

Based on all these, my conclusion now is that this DDL statement is meant to be executed from a SQL client when Spark acts a database to it. That means if you connect Spark from SQL workbench or any other SQL editors using SparkThrift server, you could probably invoke this. If that is successful, you could also try to do it programmatically by using a JDBC/ODBC driver which then connects to Thrift server.