如何解决SBT的库依赖关系警告？

Question

I'm trying to build a "hello world"-esque app that uses Spark streaming to stream data from a Kafka broker (this works), filters/processes this data, and pushes it to a (local) web browser using the Scalatra web framework and its supported web sockets functionality from Atmosphere. The Kafka/Spark chunk works independently, and the Scalatra/Atmosphere chunk also works independently. It's when I try to bring the two halves together that I run into issues with library dependencies.

The real question: how do I go about selecting library versions for which Spark will play nice with Scalatra?

A bare bones Scalatra/Atmosphere app works fine as follows:

organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.12.2"

val ScalatraVersion = "2.5.+"

libraryDependencies ++= Seq(
  "org.json4s"                  %% "json4s-jackson"      % "3.5.2",
  "org.scalatra"                %% "scalatra"            % ScalatraVersion,
  "org.scalatra"                %% "scalatra-scalate"    % ScalatraVersion,
  "org.scalatra"                %% "scalatra-specs2"     % ScalatraVersion    % "test",
  "org.scalatra"                %% "scalatra-atmosphere" % ScalatraVersion,
  "org.eclipse.jetty"           %  "jetty-webapp"        % "9.4.6.v20170531"  % "provided",
  "javax.servlet"               %  "javax.servlet-api"   % "3.1.0"            % "provided"
)

enablePlugins(JettyPlugin)

But if I add new dependencies for Spark and Spark streaming, and knock the Scala version down to 2.11 (required for Spark-Kafka streaming):

organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.11.8"

val ScalatraVersion = "2.5.+"
val SparkVersion = "2.2.0"

libraryDependencies ++= Seq(
  "org.json4s"                  %% "json4s-jackson"      % "3.5.2",
  "org.scalatra"                %% "scalatra"            % ScalatraVersion,
  "org.scalatra"                %% "scalatra-scalate"    % ScalatraVersion,
  "org.scalatra"                %% "scalatra-specs2"     % ScalatraVersion    % "test",
  "org.scalatra"                %% "scalatra-atmosphere" % ScalatraVersion,
  "org.eclipse.jetty"           %  "jetty-webapp"        % "9.4.6.v20170531"  % "provided",
  "javax.servlet"               %  "javax.servlet-api"   % "3.1.0"            % "provided"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % SparkVersion,
  "org.apache.spark" %% "spark-streaming" % SparkVersion,
  "org.apache.spark" %% "spark-streaming-kafka-0-8" % SparkVersion
)

enablePlugins(JettyPlugin)

The code compiles, but I get SBT's eviction warning:

[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * org.json4s:json4s-jackson_2.11:3.2.11 -> 3.5.3
[warn] Run 'evicted' to see detailed eviction warnings

Then finally, when Jetty tries to run the web server, it fails with this error:

WARN:oejuc.AbstractLifeCycle:main: FAILED org.eclipse.jetty.annotations.ServletContainerInitializersStarter@53fb3dab: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider

How do I get to the bottom of this? I'm new to the Scala world, and the intricacies of dependencies are blowing my mind.

Answer 1

One way to remove the eviction warning is to add the library dependency with the required version using dependencyOverrides

try to add the following in your SBT file and re-build the application

dependencyOverrides += "org.json4s" % "json4s-jackson_2.11" % "3.5.3"

Check SBT documentation here