[英]In sbt, how can we specify the version of hadoop on which spark depends?
Well I have a sbt project which uses spark and spark sql, but my cluster uses hadoop 1.0.4 and spark 1.2 with spark-sql 1.2, currently my build.sbt looks like this:好吧,我有一个使用 spark 和 spark sql 的 sbt 项目,但是我的集群使用 hadoop 1.0.4 和 spark 1.2 和 spark-sql 1.2,目前我的 build.sbt 看起来像这样:
libraryDependencies ++= Seq(
"com.datastax.cassandra" % "cassandra-driver-core" % "2.1.5",
"com.datastax.cassandra" % "cassandra-driver-mapping" % "2.1.5",
"com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.1",
"org.apache.spark" % "spark-core_2.10" % "1.2.1",
"org.apache.spark" % "spark-sql_2.10" % "1.2.1",
)
It turns out that I am running the app with hadoop 2.2.0, but I wish to see hadoop-*-1.0.4 in my dependencies.事实证明,我正在使用 hadoop 2.2.0 运行该应用程序,但我希望在我的依赖项中看到 hadoop-*-1.0.4。 What would I do please?
请问我该怎么办?
You can exclude the dependency from Spark to hadoop, and add an explicit one with the version you need, something along those lines:您可以排除 Spark 到 hadoop 的依赖项,并添加一个具有您需要的版本的显式依赖项,类似以下内容:
libraryDependencies ++= Seq(
"com.datastax.cassandra" % "cassandra-driver-core" % "2.1.5",
"com.datastax.cassandra" % "cassandra-driver-mapping" % "2.1.5",
"com.datastax.spark" % "spark-cassandra-connector" %% "1.2.1",
"org.apache.spark" % "spark-sql_2.10" % "1.2.1" excludeAll(
ExclusionRule("org.apache.hadoop")
),
"org.apache.hadoop" % "hadoop-client" % "2.2.0"
)
You probably do not need the dependency to spark-core
since spark-sql
should transitively bring it to you.您可能不需要对
spark-core
的依赖,因为spark-sql
应该可以传递给您。
Also, watch out that spark-cassandra-connector
probably also depends on spark, which could again transitively bring back hadoop => you might need to add an exclusion rule there as well.另外,请注意
spark-cassandra-connector
可能也依赖于 spark,这可能会再次传递回 hadoop => 您可能还需要在那里添加排除规则。
Last note: an excellent tool for investigating which dependency comes from where is https://github.com/jrudolph/sbt-dependency-graph最后说明:用于调查哪个依赖项来自何处的优秀工具是https://github.com/jrudolph/sbt-dependency-graph
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.