如何使用Java从Apache Spark 2.X连接到Phoenix

Question

Surprisingly , couldn't find any up to date JAVA document in the web for this . The 1 or 2 examples in the entire World Wild Web is too old. I came up with the following which fails with error ' Module not Found org.apache.phoenix.spark ' , but That module is part of the Jar for Sure . I don't think following approach is right because it is copy - paste from different examples, and loading a module like this is a bit anti pattern , as we already have the package as part of the jar. Please show me the right way.

Note- Please do Scala or Phython example , They are easily available over net,

public class ECLoad {
    public static void main(String[] args){
        //Create a SparkContext to initialize
        String warehouseLocation = new File("spark-warehouse").getAbsolutePath();
        SparkSession spark = SparkSession
                .builder()
                .appName("ECLoad")
                .master("local")
                .config("spark.sql.warehouse.dir", warehouseLocation)
                .getOrCreate();

        spark.conf().set("spark.testing.memory", "2147480000");         // if you face any memory issue
        Dataset<Row> df = spark.sqlContext().read().format("org.apache.phoenix.spark.*").option("table",
                "CLINICAL.ENCOUNTER_CASES").option("zkUrl", "localhost:2181").load();
        df.show();
    }

} 

I'm trying to run it as

spark-submit --class "encountercases.ECLoad" --jars phoenix-spark-5.0.0-HBase-2.0.jar,phoenix-core-5.0.0-HBase-2.0.jar --master local ./PASpark-1.0-SNAPSHOT.jar

and I get following error -

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration

I see required jars are already at the suggested path and hbase-site.xml symlink exixsts.

Answer 1

Before getting phoenix working with spark, you will need to setup the environment for spark so that it knows how to access phoenix/hbase.

1. First create a symbolic link to hbase-site.xml ln -s /etc/hbase/conf/hbase-site.xml /etc/spark2/conf/hbase-site.xml Alternatively you can add this file while creating spark session or in spark defaults.

2. You will need to add jars under /usr/hdp/current/phoenix-client/ to driver as well as executor class path. Parameters to be set: spark.driver.extraClassPath and spark.executor.extraClassPath

3. This step is trivial and could easily be translated into java/scala/python/R, above 2 steps are critical for it to work as those setup env:

val df = sqlContext.load("org.apache.phoenix.spark",Map("table" -> "CLINICAL.ENCOUNTER_CASES", "zkUrl" -> "localhost:2181"))

Refer to: https://community.hortonworks.com/articles/179762/how-to-connect-to-phoenix-tables-using-spark2.html