添加Flink ML依賴項

Question

我正在嘗試創建一個sbt Scala項目，並從Flink添加一些代碼。我已經成功添加了大多數Flink項目，但添加了Flink-ML。

我嘗試添加不同的依賴項，錯誤如下：

sbt.build配置：

resolvers += Resolver.mavenLocal
// I have downloaded the Flink project and compiled in local.
libraryDependencies += "org.apache.flink" % "flink-ml_2.10" % "1.0-SNAPSHOT"

錯誤如下：

Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving org.scala-sbt#cache;0.13.8 ...
[info] Resolving org.scala-sbt#testing;0.13.8 ...
[info] Resolving org.scala-sbt#test-agent;0.13.8 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving org.scala-sbt#main-settings;0.13.8 ...
[info] Resolving org.scala-sbt#apply-macro;0.13.8 ...
[info] Resolving org.scala-sbt#command;0.13.8 ...
[info] Resolving org.scala-sbt#logic;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_8_2;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_9_2;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_9_3;0.13.8 ...
[error] Modules were resolved with conflicting cross-version suffixes in {file:/home/ubuntu/workspace/Flink-project-sbt/}flink-project-sbt:
[error]    org.apache.flink:flink-runtime <none>, _2.10
[error]    org.apache.flink:flink-scala <none>, _2.10
[error]    org.apache.flink:flink-optimizer <none>, _2.10
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) Conflicting cross-version suffixes in: org.apache.flink:flink-runtime, org.apache.flink:flink-scala, org.apache.flink:flink-optimizer
[error] (*:ssExtractDependencies) Conflicting cross-version suffixes in: org.apache.flink:flink-runtime, org.apache.flink:flink-scala, org.apache.flink:flink-optimizer
[error] Total time: 33 s, completed Feb 23, 2016 8:56:38 AM</pre><br/>See complete log in <a href="/home/ubuntu/.IntelliJIdea15/system/log/sbt.last.log">/home/ubuntu/.IntelliJIdea15/system/log/sbt.last.log</a>

我也嘗試過flink ml 2.11

libraryDependencies += "org.apache.flink" % "flink-ml_2.11" % "0.10.1"

錯誤如下：

Error:Error while importing SBT project:<br/>...<br/><pre>[error]    org.apache.flink:flink-scala <none>, _2.11
[error]    org.apache.flink:flink-optimizer <none>, _2.11
[error]    org.apache.flink:flink-shaded-hadoop2 <none>, _2.11
[error]    org.clapper:grizzled-slf4j _2.10, _2.11
[error]    com.typesafe.akka:akka-actor _2.10, _2.11
[error]    com.twitter:chill _2.10, _2.11
[error]    com.twitter:bijection-core _2.10, _2.11
[error]    com.twitter:bijection-avro _2.10, _2.11
[error]    org.apache.flink:flink-core <none>, _2.11
[error]    com.twitter:chill-bijection _2.10, _2.11
[error]    com.twitter:chill-avro _2.10, _2.11
[error]    com.github.scopt:scopt _2.10, _2.11
[error]    com.typesafe.akka:akka-remote _2.10, _2.11
[error]    org.apache.flink:flink-java <none>, _2.11
[error]    com.typesafe.akka:akka-slf4j _2.10, _2.11
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) Conflicting cross-version suffixes in: org.apache.flink:flink-runtime, org.apache.flink:flink-scala, org.apache.flink:flink-optimizer, org.apache.flink:flink-shaded-hadoop2, org.clapper:grizzled-slf4j, com.typesafe.akka:akka-actor, com.twitter:chill, com.twitter:bijection-core, com.twitter:bijection-avro, org.apache.flink:flink-core, com.twitter:chill-bijection, com.twitter:chill-avro, com.github.scopt:scopt, com.typesafe.akka:akka-remote, org.apache.flink:flink-java, com.typesafe.akka:akka-slf4j
[error] (*:ssExtractDependencies) Conflicting cross-version suffixes in: org.apache.flink:flink-runtime, org.apache.flink:flink-scala, org.apache.flink:flink-optimizer, org.apache.flink:flink-shaded-hadoop2, org.clapper:grizzled-slf4j, com.typesafe.akka:akka-actor, com.twitter:chill, com.twitter:bijection-core, com.twitter:bijection-avro, org.apache.flink:flink-core, com.twitter:chill-bijection, com.twitter:chill-avro, com.github.scopt:scopt, com.typesafe.akka:akka-remote, org.apache.flink:flink-java, com.typesafe.akka:akka-slf4j
[error] Total time: 21 s, completed Feb 23, 2016 8:59:12 AM</pre><br/>See complete log in <a href="/home/ubuntu/.IntelliJIdea15/system/log/sbt.last.log">/home/ubuntu/.IntelliJIdea15/system/log/sbt.last.log</a>

我的sbt文件內容是：

name := "Flink-project-sbt"

version := "1.0"

scalaVersion := "2.10.6"



libraryDependencies ++= Seq("org.apache.flink" % "flink-scala" % "0.10.2")
libraryDependencies ++= Seq("org.apache.flink" % "flink-streaming-scala" % "0.10.2", "org.apache.flink" % "flink-clients" % "0.10.2")

// Table API dependencies
libraryDependencies += "org.apache.flink" % "flink-table" % "0.10.1"

// Machine Learning
libraryDependencies += "org.apache.flink" % "flink-ml_2.11" % "0.10.1"
// Machine Learning
resolvers += Resolver.mavenLocal
//libraryDependencies += "org.apache.flink" % "flink-ml_2.10" % "1.0-SNAPSHOT"

// Gelly
libraryDependencies += "org.apache.flink" % "flink-gelly-scala" % "0.10.1"

// for more dependencies see below url
// http://mvnrepository.com/artifact/org.apache.flink

fork in run := true

flink ml中是否有任何與scala 2.10兼容的版本？

謝謝！

Answer 1

問題在於您正在混合使用0.10.2版本和1.0-SNAPSHOT版本。 在后一個版本中，我們適當地介紹了Scala版本后綴_2.10和_2.11 。 此外，我們已糾正了一些依賴性問題，這些問題阻止了sbt程序集插件構建胖子。 因此，我強烈建議使用當前的主版本1.0-SNAPSHOT 。

可能的sbt文件可能看起來像

resolvers in ThisBuild ++= Seq(Resolver.mavenLocal)

name := "flinkMLTest"

version := "0.1-SNAPSHOT"

organization := "org.example"

scalaVersion in ThisBuild := "2.10.6"

val flinkVersion = "1.0-SNAPSHOT"

val flinkDependencies = Seq(
  "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-ml" % flinkVersion)

lazy val root = (project in file(".")).
  settings(
    libraryDependencies ++= flinkDependencies
  )

// make run command include the provided dependencies
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))

還有一個用於Flink作業的新giter8模板。 如果安裝了giter8 ，則只需發出以下命令即可使用Scala和SBT生成Flink項目。

g8 tillrohrmann/flink-project

Answer 2

1. 您應該使用Flink 0.10.2或Flink 1.0-SNAPSHOT，但不要同時使用兩者
2. 在Flink 0.10.2中沒有Scala版本后綴（即_2.10或_2.11 ）
3. Flink 0.10.2是使用Scala 2.10構建的（如果您想要Scala 2.11，則需要根據源代碼自行構建Flink）
4. Flink 1.0將附帶許多模塊的Scale版本后綴（例如， flink-scala變為flink-scala_2.10和flink-scala_2.11 ）。 這些后綴已經在當前的master中使用

因此，您需要確定要使用的Flink版本（對於當前的主設備是Scala版本），並以統一的方式指定依賴項。