Scala/Spark：引起：java.lang.ClassNotFoundException：org.apache.spark.Logging

Question

我對 Scala 和 Spark 很陌生。 試圖修復我的 spark/scala 開發設置。 我對版本和缺少的罐子感到困惑。 我在 stackoverflow 上搜索過，但仍然卡在這個問題上。 也許缺少某些東西或配置錯誤。

運行命令：

me@Mycomputer:~/spark-2.1.0$ bin/spark-submit --class ETLApp /home/me/src/etl/target/scala-2.10/etl-assembly-0.1.0.jar

輸出：

...
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
...
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging

構建.sbt：

name := "etl"

version := "0.1.0"
scalaVersion := "2.10.5"
javacOptions ++= Seq("-source", "1.8", "-target", "1.8")
mainClass := Some("ETLApp")

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % "1.5.2";
libraryDependencies += "com.datastax.spark"  %% "spark-cassandra-connector" % "1.5.0-M2";
libraryDependencies += "org.apache.curator" % "curator-recipes" % "2.6.0"
libraryDependencies += "org.apache.curator" % "curator-test" % "2.6.0"
libraryDependencies += "args4j" % "args4j" % "2.32"

java -version java 版本“1.8.0_101”

Scala 版本 2.10.5

火花版本 2.1.0

歡迎任何提示。 謝謝

Answer 1

在這種情況下，您的 jar 在提交給 spark 時必須帶上所有依賴類。

在 maven 中，這可以通過程序集插件和 jar-with-dependencies 描述符來實現。 使用 sbt 快速谷歌發現這個： https : //github.com/sbt/sbt-assembly

Answer 2

您可以按如下方式更改 build.sbt：

name := "etl"

version := "0.1.0"

scalaVersion := "2.10.5"

scalacOptions ++= Seq("-deprecation",
  "-feature",
  "-Xfuture",
  "-encoding",
  "UTF-8",
  "-unchecked",
  "-language:postfixOps")

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.2" % Provided,
"org.apache.spark" %% "spark-sql" % "1.5.2" % Provided,
"org.apache.spark" %% "spark-streaming" % "1.5.2" %  Provided,
"org.apache.spark" %% "spark-streaming-kafka" % "1.5.2" % Provided,
"com.datastax.spark"  %% "spark-cassandra-connector" % "1.5.0-M2",
"org.apache.curator" % "curator-recipes" % "2.6.0",
"org.apache.curator" % "curator-test" % "2.6.0",
"args4j" % "args4j" % "2.32")

mainClass in assembly := Some("your.package.name.ETLApp")

assemblyJarName in assembly := s"${name.value}-${version.value}.jar"

assemblyMergeStrategy in assembly := {
  case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
  case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard
  case "reference.conf" => MergeStrategy.concat
  case x: String if x.contains("UnusedStubClass.class") => MergeStrategy.first
  case _ => MergeStrategy.first
}

將 sbt-assembly 插件添加到項目根目錄下項目目錄下的 plugins.sbt 文件中。 在項目根目錄下的終端（Linux）或 CMD（Windows）中運行sbt assembly將為您下載所有依賴項並創建一個 U