从Oozie中安排的Spark Scala向Hbase写入时出现问题

Question

Hi From our spark scala app, we are connecting to hbase and writing. When we run the jar through spark-submit it works like a charm.

<action name="spark-action">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>yarn-cluster</master>
            <mode>cluster</mode>
            <name>Hbase-Test</name>
              <class>org.sample.ConnectorTest</class>
            <jar>hdfs://nameservice1/app/MyhbaseConnector.jar</jar>
              <spark-opts>--jars ${sparkLib} --files ${files} --driver-class-path ${driverClassPath}
              </spark-opts>
              <arg>testValue</arg>
        </spark>
        <ok to="mail"/>
        <error to="Kill"/>
    </action>

But when the same is scheduled in oozie workflow in a spark-action we are getting the below exception. We are also passing some spark opts to the action.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 13 more 

Answer 1

Download:

https://raw.githubusercontent.com/swordsmanliu/SparkStreamingHbase/master/lib/spark-core_2.11-1.5.2.logging.jar

And run:

spark-submit --jars ./spark-core_2.11-1.5.2.logging.jar ...

That is because org.apache.spark.Logging had been canceled at spark 1.6+