Spark流：非结构化记录

Question

We are consuming data from EventHub using streaming. Incoming stream contains JSON records of various types (around 400 different types)

Each record will be categorized using ProductId property.

example (Incoming stream of records):

record1 - { productId: 101, colA: "some-val", colB: "some-val" }
record2 - { productId: 104, colC: "some-val", colD: "some-val" }
record3 - { productId: 202, colA: "some-val", colD: "some-val", colF: "some-val" }
record3 - { productId: 342, colH: "some-val", colJ: "some-val", colK: "some-val" }

Number of properties in each record varies, but record having similar productId will have exact same number of properties.

ProductId ranges from (1 - 400), number of properties in record will be upto 50.

I want to read above stream of JSON record & write to different parquet/delta locations like

    Location(Delta/Parquet)             Records
    -----------------------------------------------------------------
    /mnt/product-101        Contains all records with productId - 101
    /mnt/product-104        Contains all records with productId - 104
    /mnt/product-202        Contains all records with productId - 202
    /mnt/product-342        Contains all records with productId - 342

1) How to create DataFrame/Dataset from stream containing different types of record?

2) Will it be possible using single spark stream & writing to different delta/parquet location ?

Answer 1

Please note the using this method should work however it will generate lots of sparse data.

First create a StructType with all the columns combined.

  val schema = new StructType().add("productId", LongType)
  .add("colA", StringType).add("colB", StringType)
  .add("colC", StringType)....

Then create a stream using this schema and the from_json function.

val df1 = df.select(from_json('value, schema).alias("tmp"))
.select("tmp.*")

Finally use partitionBy to write partitioned parquet files.

val query1 = df1 
.writeStream 
.format("parquet") 
.option("path", "/mnt/product/") 
.option("checkpointLocation","/tmp/checkpoint")
.partitionBy("productId").start()

This will generate rows with all the columns. The columns that weren't originally in the json input will be marked as null. Parquet supports writing nulls. But it would be better if you filter them first.