Structured streaming debugging input

Question

Is there a way for me to print out the incoming data? For eg I have a readStream on a folder looking for JSON files, however there seems to be an issue as I am seeing 'nulls' in the aggregation output.

val schema = StructType(
      StructField("id", LongType, false) ::
      StructField("sid", IntegerType, true) ::
      StructField("data", ArrayType(IntegerType, false), true) :: Nil)

val lines = spark.
      readStream.
      schema(schema).
      json("in/*.json")

val top1 = lines.groupBy("id").count()

val query = top1.writeStream
      .outputMode("complete")
      .format("console")
      .option("truncate", "false")
      .start()

Answer 1

To print the data you can add queryName in the write stream, by using that queryName you can print.

In your Example

val query = top1.writeStream
      .outputMode("complete")
      .queryName("xyz")
      .format("console")
      .option("truncate", "false")
      .start()

run this and you can display data by using SQL query

%sql select * from xyz 

or you can Create Dataframe

val df = spark.sql("select * from xyz")