通过PySpark将实时Kafka数据吸收到HBase中-java.lang.NoSuchMethodError：hbase.client.Put.add（[B [B [B）Lorg / apache / hadoop / hbase / client / Put;

Question

I try to configure Real-Time Kafka Data Ingestion into HBase via PySpark in accordance to this tutorial . I have a problem with the code shown below. At the moment I just try to add data to the Hbase table in the simplest possible way like this:

def SaveToHBase(rdd):

#     print("=====Pull from Stream=====")

    if not rdd.isEmpty():

        host = 'myhost:2182'
        table = 'logs'  
        keyConv = "org.apache.spark.examples.pythonconverters.StringToImmutableBytesWritableConverter"  
        valueConv = "org.apache.spark.examples.pythonconverters.StringListToPutConverter"  
        conf = {"hbase.zookeeper.quorum": host,
                "hbase.zookeeper.property.clientPort": "2182",
            "hbase.mapred.outputtable": table,
            "mapreduce.outputformat.class": "org.apache.hadoop.hbase.mapreduce.TableOutputFormat",  
            "mapreduce.job.output.key.class": "org.apache.hadoop.hbase.io.ImmutableBytesWritable",  
            "mapreduce.job.output.value.class": "org.apache.hadoop.io.Writable"}


        rdd.saveAsNewAPIHadoopDataset(conf=conf,keyConverter=keyConv,valueConverter=valueConv)


kds = KafkaUtils.createDirectStream(ssc, topic, k_params, fromOffsets=None)



###################################################################### added
parsed = kds.filter(lambda x: x != None and len(x) > 0 )
parsed = parsed.map(lambda x: x[1])
parsed = parsed.map(lambda x: (str('121323322323'),[str('121323322323'),"log","log",'content']))  

parsed.foreachRDD(SaveToHBase)


############################################################
# Start application
############################################################

runApplication(ssc, config)

It seems to me that all jars are compatible with my Hbase 2.0.2. Any ideas why I get error shown below?

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.saveAsHadoopDataset.
: org.apache.spark.SparkException: Job aborted.
    at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:100)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1081)
    at org.apache.spark.api.python.PythonRDD$.saveAsHadoopDataset(PythonRDD.scala:583)
    at org.apache.spark.api.python.PythonRDD.saveAsHadoopDataset(PythonRDD.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 72.0 failed 1 times, most recent failure: Lost task 0.0 in stage 72.0 (TID 72, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows
    at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
    at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
    at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.add([B[B[B)Lorg/apache/hadoop/hbase/client/Put;
    at org.apache.spark.examples.pythonconverters.StringListToPutConverter.convert(HBaseConverters.scala:68)
    at org.apache.spark.examples.pythonconverters.StringListToPutConverter.convert(HBaseConverters.scala:64)
    at org.apache.spark.api.python.PythonHadoopUtil$$anonfun$convertRDD$1.apply(PythonHadoopUtil.scala:181)
    at org.apache.spark.api.python.PythonHadoopUtil$$anonfun$convertRDD$1.apply(PythonHadoopUtil.scala:181)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
    at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:129)
    at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
    at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
    ... 10 more

My JARS:

os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars \
/spark/spark-streaming-kafka-0-8-assembly_2.11-2.4.0.jar,\
/spark/spark-examples_2.10-1.1.1.jar,\
/hbase/jar_files-9/* pyspark-shell'

Answer 1

I had the some problem today. Here is my solution. The error is caused by that the spark-examples-*.jar used a low-version hbase-client package (0.98), which is not compatible with hbase-client 2.* . So what we need to do is to make a little change and re-package it.

1. Download spark from http://spark.apache.org/
2. edit HBaseConverters.scale
3. mvn clean install -e -X -pl :spark-examples_2.11_hardfixed

Maybe I can attach my fixed jar here later.