ERROR when using SPARK : java.lang.NoClassDefFoundError: dispatch/Http

Question

When using spark-submit this error always happen:

ERROR Executor: Exception in task 0.0 in stage 137.0 (TID 35)
java.lang.NoClassDefFoundError: dispatch/Http$

although I have import the lib (I'm using scala) :

import dispatch._, Defaults._

Part of code :

def publishToNsq(data: String) = {
        val topic = "post_similar_items_to_elasticsearch"
        val host = "192.168.100.160:4151"
        def myRequest = url(s"$host/pub?topic=$topic")
        def myRequestAsJson = myRequest.setContentType("application/json", "UTF-8")
        def myPostWithBody = myRequestAsJson << data
        val response = Http(myPostWithBody OK as.String)
        println(response)
    }

This is part of my build.sbt :

scalaVersion := "2.10.5"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.6.1"
libraryDependencies += "org.json4s" %% "json4s-native" % "3.4.0"
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.4.0"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "1.1.4"
libraryDependencies += "net.databinder.dispatch" %% "dispatch-core" % "0.11.2"
resolvers += Resolver.mavenLocal

Any solution ?

Answer 1

依赖关系不在jar中，如果您按照java.lang.NoClassDefFoundError中的配置进行配置，则可以解决：org / apache / spark / streaming / twitter / TwitterUtils