how to visit hbase using spark 2.*

Question

I have wrote a program which visit HBase using spark 1.6 with spark-hbase-connecotr ( sbt dependency: "it.nerdammer.bigdata" % "spark-hbase-connector_2.10" % "1.0.3"). But it doesn't work when using spark 2.*. I've searched about this question and I got some concludes:

1. there are several connectors used to connect hbase using spark

   • hbase-spark. hbase-spark is provided by HBase official website . But I found it is developed on scala 2.10 and spark 1.6. The properties in the pom.xml of the project is as below:

      <properties> <spark.version>1.6.0</spark.version> <scala.version>2.10.4</scala.version> <scala.binary.version>2.10</scala.binary.version> <top.dir>${project.basedir}/..</top.dir> <avro.version>1.7.6</avro.version> <avro.mapred.classifier></avro.mapred.classifier> </properties> 

   • hbase-spark-connecotr: I visit their website and there is no information about spark 2.0. And the jar's name is spark-hbase-connector_2.10 that tell us the jar is compiled by scala 2.10 used by spark 1.* . But when I change the jar's name to spark-hbase-connector_2.11(compiled by scala 2.11,the same as spark 2.),my IEDA(a kind of IDE) tell me no jar named spark-hbase-connecotr_2.11.So there is no support for spark 2. .

   • hortonworks-spark: I 've visited their website. Some remarks said that is is not support for spark 2.*.

Do you know any jar provided by third party with full document which solve the problem? what packages should I use to connect the hbase using spark 2.* ? I appreciate you for any suggestions. Thanks!

Answer 1

我选择使用newAPIHadoopRDD访问spark中的hbase