Spark java.lang.NoClassDefFoundError中的spark-cassandra-connector错误：com / datastax / driver / core / ProtocolOptions $ Compression
[英]Error with spark-cassandra-connector in Spark java.lang.NoClassDefFoundError: com/datastax/driver/core/ProtocolOptions$Compression
问题描述
I have this error when I try to connect to cassandra with spark-cassandra-connector: 
当我尝试使用spark-cassandra-connector连接到cassandra时出现此错误：
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/driver/core/ProtocolOptions$Compression at com.datastax.spark.connector.cql.CassandraConnectorConf$.(CassandraConnectorConf.scala:112) at com.datastax.spark.connector.cql.CassandraConnectorConf$.(CassandraConnectorConf.scala) at com.datastax.spark.connector.cql.CassandraConnector$.apply(CassandraConnector.scala:192) at com.datastax.spark.connector.SparkContextFunctions.cassandraTable$default$3(SparkContextFunctions.scala:48) at main.scala.TestSpark$.main(TestSpark.scala:19) at main.scala.TestSpark.main(TestSpark.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) at org.apa
I have added the dependencies in the sbt file: 我在sbt文件中添加了依赖项：
name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "1.5.0-M2"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector-java" % "1.5.0-M2"
This is the scala program I try to execute: 这是我尝试执行的scala程序：
package main.scala
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
/**
* Created by Simo on 01.12.15.
*/
object TestSpark {
def main(args: Array[String]) {
val conf = new SparkConf(true)
.set("spark.cassandra.connection.host", "54.229.218.236")
.setAppName("Simple Application")
val sc= new SparkContext("local", "test", conf)
val rdd = sc.cassandraTable("test", "kv")
println(rdd.count)
println(rdd.first)
println(rdd.map(_.getInt("value")).sum)
}
}
And this is how i run it: 这就是我的运行方式：
$ sbt package
$ $SPARK_HOME/bin/spark-submit --class "main.scala.TestSpark" target/scala-2.11/simple-project_2.11-1.0.jar
Can you help me to understand what I'm doing wrong? 您能帮助我了解我在做什么错吗？
Thanks! 谢谢！
Edit: 编辑：
I have tried to add the Datastax driver in the dependencies list and in the spark classpath: 我试图在依赖项列表和spark类路径中添加Datastax驱动程序：
libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core" % "2.1.9"
libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-mapping" % "2.1.9"
The last error no longer appear but now I have another error: 最后一个错误不再出现，但现在我又遇到了一个错误：
Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.ObjectRef.zero()Lscala/runtime/ObjectRef;
Edit 2: Make scala 2.10.6 at compile time (same as the scala version of spark) The previous error no longer appear but i have this new error: 编辑2：在编译时使scala 2.10.6（与spark的scala版本相同）不再出现以前的错误，但是我有这个新错误：
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/util/concurrent/AsyncFunction at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFactory.scala:36) at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85) at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150) at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31) at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56) at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81) at com.datastax.spark.conne
2 个解决方案
解决方案1
2 2015-12-02 13:14:53
Finally resolved using sbt-assembly as suggested by @Odomontois 最终使用@Odomontois建议的sbt-assembly解决
This is the final build.sbt: 这是最终的build.sbt：
name := "Simple Project"
version := "1.0"
scalaVersion := "2.10.6"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" % "provided"
libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core" % "2.1.9"
libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.5.0-M2"
jarName in assembly :="my-project-assembly.jar"
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("netty", "handler", xs @ _*) => MergeStrategy.first
case PathList("netty", "buffer", xs @ _*) => MergeStrategy.first
case PathList("netty", "common", xs @ _*) => MergeStrategy.first
case PathList("netty", "transport", xs @ _*) => MergeStrategy.first
case PathList("netty", "codec", xs @ _*) => MergeStrategy.first
case PathList("META-INF", "io.netty.versions.properties") => MergeStrategy.first
case x => old(x)
}
}
解决方案2
0 2015-12-02 03:54:02
您还需要从（按照火花卡桑德拉连接器的版本）添加Datastax卡桑德拉驱动的依赖性： - https://repo1.maven.org/maven2/com/datastax/cassandra/cassandra-driver-core/
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.