[英]Spark MLlib example, NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame()
I'm following the documentation example Example: Estimator, Transformer, and Param 我正在跟踪文档示例, 例如:Estimator,Transformer和Param
And I got error msg 我收到错误信息
15/09/23 11:46:51 INFO BlockManagerMaster: Registered BlockManager Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
15/09/23 11:46:51信息BlockManagerMaster:“主”线程中注册的BlockManager异常java.lang.NoSuchMethodError:scala.reflect.api.JavaUniverse.runtimeMirror(Ljava / lang / ClassLoader;)Lscala / reflect / api / JavaUniverse $ JavaMirror; at SimpleApp$.main(hw.scala:75)
在SimpleApp $ .main(hw.scala:75)
And line 75 is the code "sqlContext.createDataFrame()": 第75行是代码“ sqlContext.createDataFrame()”:
import java.util.Random
import org.apache.log4j.Logger
import org.apache.log4j.Level
import scala.io.Source
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.rdd._
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.param.ParamMap
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.mllib.recommendation.{ALS, Rating, MatrixFactorizationModel}
import org.apache.spark.sql.Row
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions._
object SimpleApp {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Simple Application").setMaster("local[4]");
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
val training = sqlContext.createDataFrame(Seq(
(1.0, Vectors.dense(0.0, 1.1, 0.1)),
(0.0, Vectors.dense(2.0, 1.0, -1.0)),
(0.0, Vectors.dense(2.0, 1.3, 1.0)),
(1.0, Vectors.dense(0.0, 1.2, -0.5))
)).toDF("label", "features")
}
}
And my sbt is like below: 我的sbt如下所示:
lazy val root = (project in file(".")).
settings(
name := "hello",
version := "1.0",
scalaVersion := "2.11.4"
)
libraryDependencies ++= {
Seq(
"org.apache.spark" %% "spark-core" % "1.4.1" % "provided",
"org.apache.spark" %% "spark-sql" % "1.4.1" % "provided",
"org.apache.spark" % "spark-hive_2.11" % "1.4.1",
"org.apache.spark" % "spark-mllib_2.11" % "1.4.1" % "provided",
"org.apache.spark" %% "spark-streaming" % "1.4.1" % "provided",
"org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.4.1" % "provided"
)
}
I tried to search around and found this post which is very similar to my issue, and I tried to change my sbt setting for spark versions (spark-mllib_2.11 to 2.10, and spark-1.4.1 to 1.5.0), but it came even more dependency conflicts. 我试图四处搜索,发现这篇文章与我的问题非常相似,并且我尝试将我的sbt设置更改为spark版本(将spark-mllib_2.11更改为2.10,将spark-1.4.1更改为1.5.0),但是它带来了更多的依赖冲突。
My intuition is it's some version problem but cannot figure it out myself, could anyone please help? 我的直觉是这是某个版本问题,但我自己无法解决,有人可以帮忙吗? thanks a lot.
非常感谢。
It's working now for me, and just for the record, referencing @MartinSenne answer. 它现在对我有用,仅作记录用途,引用@MartinSenne答案。
what I did is as below: 我所做的如下:
@note: @注意:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.