Livy不在Hortonworks的PySpark和Spark工作

Question

我通过livy启动了一个REST spark会话，如下所示：

curl -X POST --data '{"kind": "spark"}' -H "Content-Type: application/json" localhost:8998/sessions

{
"from": 0,
"sessions": [
    {
        "id": 0,
        "kind": "spark",
        "log": [
            "16/07/18 05:05:44 INFO ScalatraListener: The cycle class name from the config: com.cloudera.livy.repl.ScalatraBootstrap",
            "16/07/18 05:05:44 INFO ScalatraListener: Initializing life cycle class: ScalatraBootstrap"
        ],
        "proxyUser": null,
        "state": "starting"
    }
],
"total": 1
}

但是我使用的是Hortonworks而不是Cloudera，我根据http://hortonworks.com/apache/zeppelin/#section_3配置了设置：

export SPARK_HOME=/usr/hdp/current/spark-client
export JAVA_HOME=/usr/jdk64/jdk1.8.0_60
export PATH=/usr/jdk64/jdk1.8.0_60/bin:$PATH
export HADOOP_CONF_DIR=/etc/hadoop/conf
export LIVY_SERVER_JAVA_OPTS="-Xmx2g"

我最终得到了这个错误：

{
    "from": 0,
    "sessions": [
        {
            "id": 0,
            "kind": "spark",
            "log": [
                "\tat com.cloudera.livy.repl.scalaRepl.SparkInterpreter.execute(SparkInterpreter.scala:111)",
                "\tat com.cloudera.livy.repl.Session.com$cloudera$livy$repl$Session$$executeCode(Session.scala:98)",
                "\tat com.cloudera.livy.repl.Session$$anonfun$3.apply(Session.scala:73)",
                "\tat com.cloudera.livy.repl.Session$$anonfun$3.apply(Session.scala:73)",
                "\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)",
                "\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)",
                "\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)",
                "\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)",
                "\tat java.lang.Thread.run(Thread.java:745)",
                "16/07/18 05:25:13 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down."
            ],
            "proxyUser": null,
            "state": "error"
        }
    ],
    "total": 1
}

我怀疑我的pyspark和spark工作没有工作，因为他们一直指着Cloudera的课而不是Hortonworks。

它可能是由于配置，因为我在https://github.com/cloudera/hue/blob/master/apps/spark/java/bin/livy-repl中看到该类默认为com。 cloudera.hue.livy.repl.Main，但我现在对Livy并不熟悉，不知道这是不是真正的bug。

如何确保Livy通过Hortonworks与Apache Spark合作？

Answer 1

我正在使用HDP 2.4中的Spark，这些是我所做的配置更改

加

spark.master  yarn-cluster

在'/usr/hdp/current/spark-client/conf/spark-defaults.conf'中

livy-env.sh

添加这些条目

export SPARK_HOME=/usr/hdp/current/spark-client   
export HADOOP_HOME=/usr/hdp/current/hadoop-client/bin/    
export HADOOP_CONF_DIR=/etc/hadoop/conf    
export SPARK_CONF_DIR=$SPARK_HOME/conf    
export LIVY_LOG_DIR=/jobserver-livy/logs    
export LIVY_PID_DIR=/jobserver-livy    
export LIVY_MAX_LOG_FILES=10    
export HBASE_HOME=/usr/hdp/current/hbase-client/bin

livy.conf

在文件末尾添加此条目

livy.rsc.rpc.server.address=<loop back ip address>