如何在结构化流中从具有不同模式（具有一些可选对象）的 kafka 主题中读取数据

Question

i have data coming in kafka topic which has an optional object, and since its optional i am missing those records when reading with a defined schema

ex:

schema i have:

val schema =new StructType()
      .add("obj_1", new ArrayType(
        new StructType(
          Array(
            StructField("field1",StringType),
            StructField("field2",StringType),
            StructField("obj_2",new ArrayType(
              new StructType(
                Array(
                  StructField("field3",StringType),
                  StructField("field4",LongType),
                  StructField("obj_3",new ArrayType(
                    new StructType(
                      Array(
                        StructField("field5",StringType),
                        StructField("field6",StringType),
                      )
                    ),containsNull = true
                  )
                  )
                )
              ),containsNull = true
            )),
            StructField("field7",StringType),
            StructField("field8",StringType))), containsNull = true))
   

when publishing data to this topic we sometimes will not send obj_3 based on some conditions.

so when reading the topic and mapping it to above schema we are missing data which will not have those obj_3 and will only contain data with that obj_3 present.

how to read the data which will not have obj_3 sometime.

sample code:

 val df = spark.readStream
        .format("kafka")
        .option("kafka.bootstrap.servers","bootstrap.servers")
        .option("subscribe", "topic.name")
        .option("startingOffsets", "offset.reset)
        .option("failOnDataLoss","true")
        .load()

      val cast = df.selectExpr( "CAST(value AS STRING)")
        .as[( String)]

val resultedDf = cast.select(from_json(col("value"), schema).as("newDF"))

 val finalDf = resultedDf.select(col("newDF.*"))

Answer 1

You could either

• use a flag (eg called "obj3flag" within a JSON structure) in the Key of the Kafka message that tells your structured streaming job if the obj_3 is existing in the Kafka Value and then choose either the one or the other schema to parse the json string. Something like:

import org.apache.spark.sql.funtions._
val schemaWithObj3 = ...
val schemaWithOutObj3 = ...

val cast = df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
        .as[(String, String)]

val resultedDf = cast
  .withColumn("obj_3_flag", get_json_object(col("key"), "$.obj3flag"))
  .withColumn("data", when(col("obj3_flag") === lit(1), from_json(col("value"), schemaWithObj3).otherwise(from_json(col("value"), schemaWithOutObj3))) 
  .select(col("data.*"))

• do a string search on "obj_3" in the Kafka value (casted as string) and if the string is found apply one or the other schema to parse the json. The code will look very similar to the one for the other option.

Please note, that I have written the code on my mobile, so you may find some syntax issues. However, hope the idea gets across.