对象StreamingContext不是包org.apache.spark的成员[错误]导入org.apache.spark.StreamingContext

Question

I am trying to run nc wordcount program in spark on SBT, and i am getting below error with log my spark version :- 1.6.3 and scala version is 2.10.0

  [warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn]  * commons-net:commons-net:2.2 is selected over 3.1
[warn]      +- org.apache.spark:spark-core_2.10:1.6.3             (depends on 2.2)
[warn]      +- org.apache.hadoop:hadoop-common:2.2.0              (depends on 3.1)
[warn]  * com.google.guava:guava:14.0.1 is selected over 11.0.2
[warn]      +- org.apache.curator:curator-recipes:2.4.0           (depends on 14.0.1)
[warn]      +- org.tachyonproject:tachyon-client:0.8.2            (depends on 14.0.1)
[warn]      +- org.apache.curator:curator-client:2.4.0            (depends on 14.0.1)
[warn]      +- org.tachyonproject:tachyon-underfs-hdfs:0.8.2      (depends on 14.0.1)
[warn]      +- org.apache.curator:curator-framework:2.4.0         (depends on 14.0.1)
[warn]      +- org.tachyonproject:tachyon-underfs-s3:0.8.2        (depends on 14.0.1)
[warn]      +- org.tachyonproject:tachyon-underfs-local:0.8.2     (depends on 14.0.1)
[warn]      +- org.apache.hadoop:hadoop-hdfs:2.2.0                (depends on 11.0.2)
[warn]      +- org.apache.hadoop:hadoop-common:2.2.0              (depends on 11.0.2)
[warn]  * com.google.code.findbugs:jsr305:1.3.9 is selected over 2.0.1
[warn]      +- com.google.guava:guava:11.0.2                      (depends on 1.3.9)
[warn]      +- org.apache.spark:spark-core_2.10:1.6.3             (depends on 1.3.9)
[warn]      +- org.apache.spark:spark-unsafe_2.10:1.6.3           (depends on 1.3.9)
[warn]      +- org.apache.spark:spark-network-common_2.10:1.6.3   (depends on 1.3.9)
[warn]      +- com.fasterxml.jackson.module:jackson-module-scala_2.10:2.4.4 (depends on 2.0.1)
[warn] Run 'evicted' to see detailed eviction warnings
[info] Compiling 1 Scala source to /home/training/Desktop/SBT/sbt/bin/sparknc/target/scala-2.10/classes ...
[error] /home/training/Desktop/SBT/sbt/bin/sparknc/src/main/scala/sparkstreaming.scala:2:8: object StreamingContext is not a member of package org.apache.spark
[error] import org.apache.spark.StreamingContext
[error]        ^
[error] /home/training/Desktop/SBT/sbt/bin/sparknc/src/main/scala/sparkstreaming.scala:6:56: value setApplication is not a member of org.apache.spark.SparkConf
[error] val mysparkconf= new SparkConf().setMaster("local[2]").setApplication("My networking application")
[error]                                                        ^
[error] /home/training/Desktop/SBT/sbt/bin/sparknc/src/main/scala/sparkstreaming.scala:7:27: not found: type StreamingContext
[error] val streamingcontext= new StreamingContext(mysparkconf, seconds(2))
[error]                           ^

[error] three errors found [error] (Compile / compileIncremental) Compilation failed

Answer 1

There is need to add the exact dependency of spark-streaming into your build.sbt file.

Set scala version as 2.10.5

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.3" % "provided"

Just suggestion use latested spark 2.3.1,with new features most of the dependency issues resolved

Adding a link for fixed issues spark docs