Spark流式弹性搜索依赖关系

Question

我在Elasticsearch指南中描述了在Scala中尝试Spark＆Elasticsearch集成

编译时我遇到依赖项问题：

[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: cascading#ing-local;2.5.6: not found
[error] unresolved dependency: clj-time#clj-time;0.4.1: not found
[error] unresolved dependency: compojure#compojure;1.1.3: not found
[error] unresolved dependency: hiccup#hiccup;0.3.6: not found
[error] unresolved dependency: ring#ring-devel;0.3.11: not found
[error] unresolved dependency: ring#ring-jetty-adapter;0.3.11: not found
[error] unresolved dependency: com.twitter#carbonite;1.4.0: not found
[error] unresolved dependency: cascading#cascading-hadoop;2.5.6: not found
[error] Total time: 86 s, completed 19 nov. 2014 08:42:58

我的build.sbt文件看起来像这样

name := "twitter-sparkstreaming-elasticsearch"

version := "0.0.1"

scalaVersion := "2.10.4"

// additional libraries
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.1.0",
  "org.apache.spark" %% "spark-streaming" % "1.1.0",
  "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",
  "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0"
)

救命？ 谢谢。

Answer 1

级联及其依赖关系在Maven中心不可用，但在他们自己的repo中（es-hadoop无法通过其pom指定）。

我使用elasticsearch-spark_2.10解决了这个问题

http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/install.html

Answer 2

Sbt无法解析某些依赖项，因为它们不是Maven存储库的一部分。 但是，你可以在clojars和conjars上找到它们。 您需要添加以下行，以便sbt可以解决它们：

resolvers += "clojars" at "https://clojars.org/repo"
resolvers += "conjars" at "http://conjars.org/repo"

而且，依赖性elasticsearch-hadoop“2.1.0”不存在（还有吗？），你应该使用“2.1.0.Beta4”（或者你读这篇文章的最新版本）

您的sbt文件应如下所示：

name := "twitter-sparkstreaming-elasticsearch"

version := "0.0.1"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "1.1.0",
    "org.apache.spark" %% "spark-streaming" % "1.1.0",
    "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",
    "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta4"
)

resolvers += "clojars" at "https://clojars.org/repo"
resolvers += "conjars" at "http://conjars.org/repo"

这已经过测试（使用spark-core 1.3.1并且没有火花流，但它应该适合你）。 希望能帮助到你。

Answer 3

这是因为级联及其依赖关系不在Maven中。 您必须添加解析器才能获得它们

将此行添加到build.sbt

resolvers += "conjars.org" at "http://conjars.org/repo"

你的build.sbt应该是这样的：

name := "twitter-sparkstreaming-elasticsearch"

version := "0.0.1"

scalaVersion := "2.10.4"

// additional libraries
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.1.0",
  "org.apache.spark" %% "spark-streaming" % "1.1.0",
  "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",
  "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0"
)

resolvers += "conjars.org" at "http://conjars.org/repo"

注意：此问题已在此处提出并关闭https://github.com/elasticsearch/elasticsearch-hadoop/issues/304 ，其解决方案与上述相同

Answer 4

"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0"尚不可用。 你可以使用"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2"

点击这里：

• http://www.mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop/2.1.0.Beta2

EDIT1

甚至在使用"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2"我也得到了同样的错误：

[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: cascading#cascading-local;2.5.6: not found
[warn]  :: clj-time#clj-time;0.4.1: not found
[warn]  :: compojure#compojure;1.1.3: not found
[warn]  :: hiccup#hiccup;0.3.6: not found
[warn]  :: ring#ring-devel;0.3.11: not found
[warn]  :: ring#ring-jetty-adapter;0.3.11: not found
[warn]  :: com.twitter#carbonite;1.4.0: not found
[warn]  :: cascading#cascading-hadoop;2.5.6: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: cascading#cascading-local;2.5.6: not found
[error] unresolved dependency: clj-time#clj-time;0.4.1: not found
[error] unresolved dependency: compojure#compojure;1.1.3: not found
[error] unresolved dependency: hiccup#hiccup;0.3.6: not found
[error] unresolved dependency: ring#ring-devel;0.3.11: not found
[error] unresolved dependency: ring#ring-jetty-adapter;0.3.11: not found
[error] unresolved dependency: com.twitter#carbonite;1.4.0: not found
[error] unresolved dependency: cascading#cascading-hadoop;2.5.6: not found
[error] Total time: 41 s, completed Nov 19, 2014 8:44:04 PM

我的build.sbt看起来像这样

name := "twitter-sparkstreaming-elasticsearch"

version := "0.0.1"

scalaVersion := "2.10.4"

// additional libraries
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.1.0",
  "org.apache.spark" %% "spark-streaming" % "1.1.0",
  "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",
  "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2",
  "org.elasticsearch" % "elasticsearch-hadoop-cascading" % "2.1.0.Beta2"
)

resolvers += "sonatype-oss" at "http://oss.sonatype.org/content/repositories/snapshots"

resolvers += "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/"

Answer 5

对于仅支持Spark，您可以使用简约二进制文件。 将以下内容添加到libraryDependencies中的build.sbt

"org.elasticsearch" % "elasticsearch-spark_2.10" % "2.1.0.Beta3"

注意：'2.10'指的是兼容的Scala版本！

并删除

"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta3"

这将避免问题中列出的未解决的deps。