Spark Streaming Kafka CreateDirectStream无法解析

Question

Need some help, please.

I am using IntelliJ with SBT to build my apps.

I'm working on an app to read a Kafka topic in Spark Streaming in order to do some ETL work on it. Unfortunately, I can't read from Kafka.

The KafkaUtils.createDirectStream isn't resolving and keeps giving me errors (CANNOT RESOLVE SYMBOL). I have done my research and it appears I have the correct dependencies.

Here is my build.sbt:

name := "ASUIStreaming"
version := "0.1"
scalacOptions += "-target:jvm-1.8"
scalaVersion := "2.11.11"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
libraryDependencies += "org.apache.kafka" %% "kafka-clients" % "0.8.2.1"
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.4"

Any suggestions? I should also mention I don't have admin access on the laptop since this is a work computer, and I am using a portable JDK and IntelliJ installation. However, my colleagues at work are in the same situation and it works fine for them.

Thanks in advance!

Answer 1

Here is the main Spark Streaming code snippet I'm using. Note : I've masked some of the confidential work data such as IP and Topic name etc.

import org.apache.kafka.clients.consumer.ConsumerRecord
import kafka.serializer.StringDecoder
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark
import org.apache.kafka.clients.consumer._
import org.apache.kafka.common.serialization.StringDeserializer
import scala.util.parsing.json._
import org.apache.spark.streaming.kafka._



object ASUISpeedKafka extends App

{
  // Create a new Spark Context
  val conf = new SparkConf().setAppName("ASUISpeedKafka").setMaster("local[*]")
  val sc = new SparkContext(conf)
  val ssc = new StreamingContext(sc, Seconds(2))

  //Identify the Kafka Topic and provide the parameters and Topic details
  val kafkaTopic = "TOPIC1"
    val topicsSet = kafkaTopic.split(",").toSet
    val kafkaParams = Map[String, String]
  (

    "metadata.broker.list" -> "IP1:PORT, IP2:PORT2",
    "auto.offset.reset" -> "smallest"
  )

  val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder]
  (
  ssc, kafkaParams, topicsSet
  )
}

Answer 2

I was able to resolve the issue. After re-creating the project and adding all dependencies again, I found out that in Intellij certain code has to be on the same line other it won't compile.

In this case, putting val kafkaParams code on the same line (instead of in a code block) solved the issue!