如何流式传输单个 kafka 主题，按键过滤到 hdfs 的多个位置？

Question

I am not being to stream my data on multiple hdfs location , which is filtered by key. So below code is not working. Please help me to find the correct way to write this code

    val ER_stream_V1 = spark
        .readStream
        .format("kafka")
        .option("kafka.bootstrap.servers", configManager.getString("Kafka.Server"))
        .option("subscribe", "Topic1")
        .option("startingOffsets", "latest")
        .option("failOnDataLoss", "false")
        .load()
val ER_stream_V2 = spark
        .readStream
        .format("kafka")
        .option("kafka.bootstrap.servers", configManager.getString("Kafka.Server"))
.option("subscribe", "Topic1")
        .option("startingOffsets", "latest")
        .option("failOnDataLoss", "false")
        .load()

        ER_stream_V1.toDF()
        .select(col("key"), col("value").cast("string"))
        .filter(col("key")==="Value1")
        .select(functions.from_json(col("value").cast("string"), Value1Schema.schemaExecution).as("value")).select("value.*")
        .writeStream
        .format("orc")
        .option("metastoreUri", configManager.getString("spark.datasource.hive.warehouse.metastoreUri"))
        .option("checkpointLocation", "/tmp/teststreaming/execution/checkpoint2005")
        .option("path", "/tmp/test/value1")
        .trigger(Trigger.ProcessingTime("5 Seconds"))
        .partitionBy("jobid")
        .start()

        ER_stream_V2.toDF()
        .select(col("key"), col("value").cast("string"))
        .filter(col("key")==="Value2")
        .select(functions.from_json(col("value").cast("string"), Value2Schema.schemaJobParameters).as("value"))
        .select("value.*")
        .writeStream
        .format("orc")
        .option("metastoreUri", configManager.getString("spark.datasource.hive.warehouse.metastoreUri"))
        .option("checkpointLocation", "/tmp/teststreaming/jobparameters/checkpoint2006")
        .option("path", "/tmp/test/value2")
        .trigger(Trigger.ProcessingTime("5 Seconds"))
        .partitionBy("jobid")
        .start()

Answer 1

You should not need two readers. Create one and filter twice. You might also want to consider startingOffsets as earliest to read existing topic data

For example.

val ER_stream = spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", configManager.getString("Kafka.Server"))
    .option("subscribe", "Topic1")
    .option("startingOffsets", "latest")  // maybe change?
    .option("failOnDataLoss", "false")
    .load()
    .toDF()
    .select(col("key").cast("string").as("key"), col("value").cast("string"))

val value1Stream = ER_stream
    .filter(col("key") === "Value1")
    .select(functions.from_json(col("value"), Value1Schema.schemaExecution).as("value"))
    .select("value.*")

val value2Stream = ER_stream
    .filter(col("key") === "Value2")
    .select(functions.from_json(col("value"), Value2Schema.schemaJobParameters).as("value"))
    .select("value.*")

value1Stream.writeStream.format("orc")
    ...
    .start()

value2Stream.writeStream.format("orc")
    ...
    .start()