Spark 从 kafka 主题流式传输数据并写入外部路径中的文本文件

Question

我想从 kafka 主题中读取数据并按键值分组，然后写入文本文件..

public static void main(String[] args) throws Exception {
        SparkSession spark=SparkSession
                .builder()
                .appName("Sparkconsumer")
                .master("local[*]")
                .getOrCreate();
        SQLContext sqlContext = spark.sqlContext();
        SparkContext context = spark.sparkContext();
        Dataset<Row>lines=spark
                .readStream()
                .format("kafka")
                .option("kafka.bootstrap.servers", "localhost:9092")
                .option("subscribe","test-topic")
                .load();
    Dataset<Row> r=  lines.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)");
        r.printSchema();
        r.createOrReplaceTempView("basicView");
        sqlContext.sql("select * from basicView")
        .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
        .writeStream()
        .outputMode("append")
        .format("console")
        .option("path","usr//path")
        .start()
        .awaitTermination(); 

Answer 1

以下几点在您的代码中具有误导性：

• 要从 Kafka 读取并写入文件，您不需要SparkContext或SQLContext ，
• 您将key和value两次转换为字符串，
• 如果要将数据存储到文件中，则输出查询的format不应为控制台。

可以在Spark Structured Streaming + Kafka 集成指南和Spark Structured Streaming Programming Guide 中查找示例

public static void main(String[] args) throws Exception {

  SparkSession spark = SparkSession
    .builder()
    .appName("Sparkconsumer")
    .master("local[*]")
    .getOrCreate();

  Dataset<Row> lines = spark
    .readStream()
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe","test-topic")
    .load();

  Dataset<Row> r = lines
    .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
    // do some more processing such as 'groupBy'
    ;

  r.writeStream
    .format("parquet")        // can be "orc", "json", "csv", etc.
    .outputMode("append")
    .option("path", "path/to/destination/dir")
    .option("checkpointLocation", "/path/to/checkpoint/dir")
    .start()
    .awaitTermination();