[英]Error in using spark-cassandra-connector: java.lang.NoSuchMethodError
[英]NoSuchMethodError from spark-cassandra-connector with assembled jar
我是Scala的新手,正在尝试建立Spark职位。 我建立了一个包含DataStax连接器的作业,并将其组装到一个胖子罐中。 当我尝试执行它时,它失败并出现java.lang.NoSuchMethodError
。 我已经破解了JAR,可以看到其中包含DataStax库。 我是否缺少明显的东西? 关于这个过程,有没有很好的教程可以看?
谢谢
控制台 $ spark-submit --class org.bobbrez.CasCountJob ./target/scala-2.11/bobbrez-spark-assembly-0.0.1.jar ks tn ...线程“ main”中的异常java.lang.NoSuchMethodError:scala .runtime.ObjectRef.zero()Lscala / runtime / ObjectRef; com.datastax.spark.connector.cql.CassandraConnector $ .com.datastax.spark.connector.cql.CassandraConnector $ .com $ datastax $ spark $ connector $ cql $ CassandraConnector $$ createSession(CassandraConnector.scala)com.datastax.spark.connector.cql.CassandraConnector $$ anonfun $ 2。 Apply(CassandraConnector.scala:148)...
build.sbt
name := "soofa-spark"
version := "0.0.1"
scalaVersion := "2.11.7"
// additional libraries
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.0" % "provided"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "1.5.0-M3"
libraryDependencies += "com.typesafe" % "config" % "1.3.0"
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
case m if m.startsWith("META-INF") => MergeStrategy.discard
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
case PathList("org", "apache", xs @ _*) => MergeStrategy.first
case PathList("org", "jboss", xs @ _*) => MergeStrategy.first
case "about.html" => MergeStrategy.rename
case "reference.conf" => MergeStrategy.concat
case _ => MergeStrategy.first
}
}
CasCountJob.scala
package org.bobbrez
// Spark
import org.apache.spark.{SparkContext, SparkConf}
import com.datastax.spark.connector._
object CasCountJob {
private val AppName = "CasCountJob"
def main(args: Array[String]) {
println("Hello world from " + AppName)
val keyspace = args(0)
val tablename = args(1)
println("Keyspace: " + keyspace)
println("Table: " + tablename)
// Configure and create a Scala Spark Context.
val conf = new SparkConf(true)
.set("spark.cassandra.connection.host", "HOSTNAME")
.set("spark.cassandra.auth.username", "USERNAME")
.set("spark.cassandra.auth.password", "PASSWORD")
.setAppName(AppName)
val sc = new SparkContext(conf)
val rdd = sc.cassandraTable(keyspace, tablename)
println("Table Count: " + rdd.count)
System.exit(0)
}
}
Spark 1.6的Cassandra连接器仍在开发中,尚未发布。
为了将Cassandra与Spark集成,您至少需要以下依赖项:-
这里提到了适当版本的Cassandra库和Spark的映射
显然,Spark 1.5的Cassandra连接器也在开发中,您可能会看到一些兼容性问题。 Cassandra连接器的最稳定版本是Spark 1.4,需要以下Jar文件:-
不用说,所有这些jar文件都应配置并可供执行者使用。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.