Spark 结构化流 -

Question

I'm trying to run the following code from IntelliJ idea to print messages from Kafka to console. But it throws the following error -

Exception in thread "main" org.apache.spark.sql.AnalysisException: Queries with streaming sources must be executed with writeStream.start();;

Stacktrace started from Dataset.checkpoint and way up. If I remove .checkpoint() then I get some other error - related to permission

17/08/02 12:10:52 ERROR StreamMetadata: Error writing stream metadata StreamMetadata(4e612f22-efff-4c9a-a47a-a36eb533e9d6) to C:/Users/rp/AppData/Local/Temp/temporary-2f570b97-ad16-4f00-8356-d43ccb7660db/metadata
java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\rp\AppData\Local\Temp\temporary-2f570b97-ad16-4f00-8356-d43ccb7660db\metadata

Source:

def main(args : Array[String]) = {
 val spark = SparkSession.builder().appName("SparkStreaming").master("local[*]").getOrCreate()
  val canonicalSchema = new StructType()
                          .add("cid",StringType)
                          .add("uid",StringType)
                          .add("sourceSystem",
                              new StructType().add("id",StringType)
                                              .add("name",StringType))
                          .add("name", new StructType()
                                        .add("firstname",StringType)
                                        .add("lastname",StringType))


val messages = spark
                    .readStream
                    .format("kafka")
                    .option("kafka.bootstrap.servers","localhost:9092")
                    .option("subscribe","c_canonical")
                    .option("startingOffset","earliest")
                    .load()
                    .checkpoint()
.select(from_json(col("value").cast("string"),canonicalSchema))
.writeStream.outputMode("append").format("console").start.awaitTermination

 }

Can anyone please help me understand where I'm doing wrong?

Answer 1

1. Structured Streaming doesn't support Dataset.checkpoint() . There is an open ticket to provide a better message or just ignore it: https://issues.apache.org/jira/browse/SPARK-20927

2. IOException probably is because you don't install cygwin on Windows.