Add Flink ML dependencies

Question

I'm trying to create a sbt Scala project adding some code from Flink.I have successfully added most of the Flink projects but Flink-ML.

I have tried to add different dependencies and the errors are as follows:

sbt.build configuration:

resolvers += Resolver.mavenLocal
// I have downloaded the Flink project and compiled in local.
libraryDependencies += "org.apache.flink" % "flink-ml_2.10" % "1.0-SNAPSHOT"

The error is as follows:

Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving org.scala-sbt#cache;0.13.8 ...
[info] Resolving org.scala-sbt#testing;0.13.8 ...
[info] Resolving org.scala-sbt#test-agent;0.13.8 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving org.scala-sbt#main-settings;0.13.8 ...
[info] Resolving org.scala-sbt#apply-macro;0.13.8 ...
[info] Resolving org.scala-sbt#command;0.13.8 ...
[info] Resolving org.scala-sbt#logic;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_8_2;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_9_2;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_9_3;0.13.8 ...
[error] Modules were resolved with conflicting cross-version suffixes in {file:/home/ubuntu/workspace/Flink-project-sbt/}flink-project-sbt:
[error]    org.apache.flink:flink-runtime <none>, _2.10
[error]    org.apache.flink:flink-scala <none>, _2.10
[error]    org.apache.flink:flink-optimizer <none>, _2.10
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) Conflicting cross-version suffixes in: org.apache.flink:flink-runtime, org.apache.flink:flink-scala, org.apache.flink:flink-optimizer
[error] (*:ssExtractDependencies) Conflicting cross-version suffixes in: org.apache.flink:flink-runtime, org.apache.flink:flink-scala, org.apache.flink:flink-optimizer
[error] Total time: 33 s, completed Feb 23, 2016 8:56:38 AM</pre><br/>See complete log in <a href="/home/ubuntu/.IntelliJIdea15/system/log/sbt.last.log">/home/ubuntu/.IntelliJIdea15/system/log/sbt.last.log</a>

I have also tried with flink ml for 2.11

libraryDependencies += "org.apache.flink" % "flink-ml_2.11" % "0.10.1"

The error is as follows:

Error:Error while importing SBT project:<br/>...<br/><pre>[error]    org.apache.flink:flink-scala <none>, _2.11
[error]    org.apache.flink:flink-optimizer <none>, _2.11
[error]    org.apache.flink:flink-shaded-hadoop2 <none>, _2.11
[error]    org.clapper:grizzled-slf4j _2.10, _2.11
[error]    com.typesafe.akka:akka-actor _2.10, _2.11
[error]    com.twitter:chill _2.10, _2.11
[error]    com.twitter:bijection-core _2.10, _2.11
[error]    com.twitter:bijection-avro _2.10, _2.11
[error]    org.apache.flink:flink-core <none>, _2.11
[error]    com.twitter:chill-bijection _2.10, _2.11
[error]    com.twitter:chill-avro _2.10, _2.11
[error]    com.github.scopt:scopt _2.10, _2.11
[error]    com.typesafe.akka:akka-remote _2.10, _2.11
[error]    org.apache.flink:flink-java <none>, _2.11
[error]    com.typesafe.akka:akka-slf4j _2.10, _2.11
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) Conflicting cross-version suffixes in: org.apache.flink:flink-runtime, org.apache.flink:flink-scala, org.apache.flink:flink-optimizer, org.apache.flink:flink-shaded-hadoop2, org.clapper:grizzled-slf4j, com.typesafe.akka:akka-actor, com.twitter:chill, com.twitter:bijection-core, com.twitter:bijection-avro, org.apache.flink:flink-core, com.twitter:chill-bijection, com.twitter:chill-avro, com.github.scopt:scopt, com.typesafe.akka:akka-remote, org.apache.flink:flink-java, com.typesafe.akka:akka-slf4j
[error] (*:ssExtractDependencies) Conflicting cross-version suffixes in: org.apache.flink:flink-runtime, org.apache.flink:flink-scala, org.apache.flink:flink-optimizer, org.apache.flink:flink-shaded-hadoop2, org.clapper:grizzled-slf4j, com.typesafe.akka:akka-actor, com.twitter:chill, com.twitter:bijection-core, com.twitter:bijection-avro, org.apache.flink:flink-core, com.twitter:chill-bijection, com.twitter:chill-avro, com.github.scopt:scopt, com.typesafe.akka:akka-remote, org.apache.flink:flink-java, com.typesafe.akka:akka-slf4j
[error] Total time: 21 s, completed Feb 23, 2016 8:59:12 AM</pre><br/>See complete log in <a href="/home/ubuntu/.IntelliJIdea15/system/log/sbt.last.log">/home/ubuntu/.IntelliJIdea15/system/log/sbt.last.log</a>

My sbt file content is:

name := "Flink-project-sbt"

version := "1.0"

scalaVersion := "2.10.6"



libraryDependencies ++= Seq("org.apache.flink" % "flink-scala" % "0.10.2")
libraryDependencies ++= Seq("org.apache.flink" % "flink-streaming-scala" % "0.10.2", "org.apache.flink" % "flink-clients" % "0.10.2")

// Table API dependencies
libraryDependencies += "org.apache.flink" % "flink-table" % "0.10.1"

// Machine Learning
libraryDependencies += "org.apache.flink" % "flink-ml_2.11" % "0.10.1"
// Machine Learning
resolvers += Resolver.mavenLocal
//libraryDependencies += "org.apache.flink" % "flink-ml_2.10" % "1.0-SNAPSHOT"

// Gelly
libraryDependencies += "org.apache.flink" % "flink-gelly-scala" % "0.10.1"

// for more dependencies see below url
// http://mvnrepository.com/artifact/org.apache.flink

fork in run := true

Is there any version in flink ml compatible with scala 2.10?

Thanks!

Answer 1

The problem is that you're mixing version 0.10.2 and version 1.0-SNAPSHOT . In the latter version we have properly introduced the Scala version suffixes _2.10 and _2.11 . Furthermore, we have corrected some dependency issues which prevented the sbt assembly plugin from building a fat jar. Therefore, I strongly recommend using the current master which is version 1.0-SNAPSHOT .

A possible sbt file could then look like

resolvers in ThisBuild ++= Seq(Resolver.mavenLocal)

name := "flinkMLTest"

version := "0.1-SNAPSHOT"

organization := "org.example"

scalaVersion in ThisBuild := "2.10.6"

val flinkVersion = "1.0-SNAPSHOT"

val flinkDependencies = Seq(
  "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-ml" % flinkVersion)

lazy val root = (project in file(".")).
  settings(
    libraryDependencies ++= flinkDependencies
  )

// make run command include the provided dependencies
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))

There is also a new giter8 template for Flink jobs. If you have giter8 installed , then you can simply issue the following command to generate a Flink project using Scala and SBT.

g8 tillrohrmann/flink-project

Answer 2

1. you should either go with Flink 0.10.2 or with Flink 1.0-SNAPSHOT but not mix both
2. in Flink 0.10.2 there are no Scala version suffixes (ie, _2.10 or _2.11 )
3. Flink 0.10.2 is built with Scala 2.10 (if you want Scala 2.11 you need to build Flink by yourself from the source code)
4. Flink 1.0 will ship with Scale version suffixes for many modules (eg., flink-scala becomes flink-scala_2.10 and flink-scala_2.11 ). Those suffixes are use in current master already

Thus, you need to decide with version of Flink (and for current master which version of Scala) you wanna use and specify the dependencies in a consisted way.