Spark 从 kafka 主题流式传输数据并写入外部路径中的文本文件

Question

I want to read a data from kafka topic and group by key values, and write into text files..

public static void main(String[] args) throws Exception {
        SparkSession spark=SparkSession
                .builder()
                .appName("Sparkconsumer")
                .master("local[*]")
                .getOrCreate();
        SQLContext sqlContext = spark.sqlContext();
        SparkContext context = spark.sparkContext();
        Dataset<Row>lines=spark
                .readStream()
                .format("kafka")
                .option("kafka.bootstrap.servers", "localhost:9092")
                .option("subscribe","test-topic")
                .load();
    Dataset<Row> r=  lines.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)");
        r.printSchema();
        r.createOrReplaceTempView("basicView");
        sqlContext.sql("select * from basicView")
        .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
        .writeStream()
        .outputMode("append")
        .format("console")
        .option("path","usr//path")
        .start()
        .awaitTermination(); 

Answer 1

Following points are misleading in your code:

• To read from Kafka and write into a file, you would not need SparkContext or SQLContext ,
• You are casting your key and value twice into a string,
• the format of your output query should not be console if you want to store the data into a file.

An example can be looked up in the Spark Structured Streaming + Kafka Integration Guide and the Spark Structured Streaming Programming Guide

public static void main(String[] args) throws Exception {

  SparkSession spark = SparkSession
    .builder()
    .appName("Sparkconsumer")
    .master("local[*]")
    .getOrCreate();

  Dataset<Row> lines = spark
    .readStream()
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe","test-topic")
    .load();

  Dataset<Row> r = lines
    .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
    // do some more processing such as 'groupBy'
    ;

  r.writeStream
    .format("parquet")        // can be "orc", "json", "csv", etc.
    .outputMode("append")
    .option("path", "path/to/destination/dir")
    .option("checkpointLocation", "/path/to/checkpoint/dir")
    .start()
    .awaitTermination();