使用java.lang.NoSuchMethodError火花读取HBase：org.apache.hadoop.mapreduce.InputSplit.getLocationInfo错误

Question

I want to use scala read Hbase by Spark, but I got error:

Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.InputSplit.getLocationInfo()[Lorg/apache/hadoop/mapred/SplitLocationInfo;

But I already add the dependencies, this problem bothers me. My environment is as follow:

• scala: 2.11.12
• Spark: 2.3.1
• HBase: maybe 2.1.0(I don't know)
• Hadoop: 2.7.2.4

And my build.sbt is：

libraryDependencies ++= Seq(
    "org.apache.spark" % "spark-core_2.11" % "2.3.1",
    "org.apache.spark" % "spark-sql_2.11" % "2.3.1",
    "org.apache.spark" % "spark-streaming_2.11" % "2.3.1",
    "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.3.1",
    "org.apache.spark" % "spark-yarn_2.11" % "2.3.1",
    "org.apache.hadoop" % "hadoop-core" % "2.6.0-mr1-cdh5.15.1",
    "org.apache.hadoop" % "hadoop-common" % "2.7.2",
    "org.apache.hadoop" % "hadoop-client" % "2.7.2",
    "org.apache.hadoop" % "hadoop-mapred" % "0.22.0",
    "org.apache.hadoop" % "hadoop-nfs" % "2.7.2",
    "org.apache.hadoop" % "hadoop-hdfs" % "2.7.2",
    "org.apache.hadoop" % "hadoop-hdfs-nfs" % "2.7.2",
    "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.7.2",
    "org.apache.hadoop" % "hadoop-mapreduce" % "2.7.2",
    "org.apache.hadoop" % "hadoop-mapreduce-client" % "2.7.2",
    "org.apache.hadoop" % "hadoop-mapreduce-client-common" % "2.7.2",
    "org.apache.hbase" % "hbase" % "2.1.0",
    "org.apache.hbase" % "hbase-server" % "2.1.0",
    "org.apache.hbase" % "hbase-common" % "2.1.0",
    "org.apache.hbase" % "hbase-client" % "2.1.0",
    "org.apache.hbase" % "hbase-protocol" % "2.1.0",
    "org.apache.hbase" % "hbase-metrics" % "2.1.0",
    "org.apache.hbase" % "hbase-metrics-api" % "2.1.0",
    "org.apache.hbase" % "hbase-mapreduce" % "2.1.0",
    "org.apache.hbase" % "hbase-zookeeper" % "2.1.0",
    "org.apache.hbase" % "hbase-hadoop-compat" % "2.1.0",
    "org.apache.hbase" % "hbase-hadoop2-compat" % "2.1.0",
    "org.apache.hbase" % "hbase-spark" % "2.1.0-cdh6.1.0"
)

I really don't know where I'm wrong, If I add wrong dependence or I need to add some new dependence, please tell me where can I download it,like: resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases"

Please help me, Thanks!

Answer 1

Can I get more details about how you are running the spark job? If you are using custom distribution such as Cloudera or Horton works, you may have to use their libraries to compile and spark-submit will use the distribution installed classpath to submit the job to cluster.

To get started, please add % provided to the library in sbt file so that it will use the particular library from the classpath of spark installation.

Answer 2

You need to fix the versions of these to match the version of Hadoop you're running, otherwise you can expect to get classpath/method issues. Specifically, your error comes from the mapreduce package

"org.apache.hadoop" % "hadoop-core" % "2.6.0-mr1-cdh5.15.1",
"org.apache.hadoop" % "hadoop-mapred" % "0.22.0",

Spark already includes most of the Hadoop ones itself, so it's not clear why you're specifying them all yourself, but at least put % "provided" on some of them

And for hbase-spark , I doubt that you want a cdh6 dependency because CDH 6 is based on Hadoop 3 libraries, not 2.7.2