[英]Spark Standalone : how to avoid sbt assembly and uber-jar?
[英]Conflicting files in uber-jar creation in SBT using sbt-assembly
我正在嘗試使用SBT編譯和打包胖罐子,並且不斷遇到以下錯誤。 我已經嘗試過使用庫依賴排除和合並的所有方法。
[trace] Stack trace suppressed: run last *:assembly for the full output.
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/me/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1 .7.10.jar:META-INF/maven/org.slf4j/slf4j-api/pom.properties
[error] /Users/me/.ivy2/cache/com.twitter/parquet-format/jars/parquet-format-2.2.0-rc1.jar:META-INF/maven/org.slf4j/slf4j-api/pom.properties
[error] Total time: 113 s, completed Jul 10, 2015 1:57:21 AM
我的build.sbt文件的當前版本如下:
import AssemblyKeys._
assemblySettings
name := "ldaApp"
version := "0.1"
scalaVersion := "2.10.4"
mainClass := Some("myApp")
libraryDependencies +="org.scalanlp" %% "breeze" % "0.11.2"
libraryDependencies +="org.scalanlp" %% "breeze-natives" % "0.11.2"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.3.1"
libraryDependencies +="org.ini4j" % "ini4j" % "0.5.4"
jarName in assembly := "myApp"
net.virtualvoid.sbt.graph.Plugin.graphSettings
libraryDependencies += "org.slf4j" %% "slf4j-api"" % "1.7.10" % "provided"
我意識到自己做錯了...我只是不知道該怎么辦。
這是處理這些合並問題的方法。
import sbtassembly.Plugin._
lazy val assemblySettings = sbtassembly.Plugin.assemblySettings ++ Seq(
publishArtifact in packageScala := false, // Remove scala from the uber jar
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", "CHANGES.txt") => MergeStrategy.first
// ...
case PathList(ps @ _*) if ps.last endsWith "pom.properties" => MergeStrategy.first
case x => old(x)
}
}
)
然后將這些設置添加到您的項目。
lazy val projectToJar = Project(id = "MyApp", base = file(".")).settings(assemblySettings: _*)
我通過從胖子罐中移除火花來運行您的程序集構建( mllib已經包含在spark中 )。
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.3.1" % "provided"
就像vitalii在評論中說的那樣,此解決方案已經在這里 。 我了解在問題上花費數小時而找不到解決方法可能會令人沮喪,但請保持友好 。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.