如何从 Spark Structured Streaming 刷新 Hive/Impala 表？

Question

currently my Spark Structured Streaming goes like this (Sink part displayed only):

    //Output aggregation query to Parquet in append mode
    aggregationQuery.writeStream
      .format("parquet")
      .trigger(Trigger.ProcessingTime("15 seconds"))
      .partitionBy("date", "hour")
      .option("path", "hdfs://<myip>:8020/user/myuser/spark/proyecto3")
      .option("checkpointLocation", "hdfs://<myip>:8020/user/myuser/spark/checkpointfolder3")
      .outputMode("append")
      .start()

The above code generates .parquet files in the directory defined by path .

I have externally defined a Impala table that reads from that path, but I need the table to be updated or refreshed after every append of parquet files.

How can this be achieved?

Answer 1

You need to update the partitions of your table after file sink.

    import spark.sql
    val query1 = "ALTER TABLE proyecto3 ADD IF NOT EXISTS PARTITION (date='20200803') LOCATION '/your/location/proyecto3/date=20200803'"
    sql(s"$query1")

    import spark.sql
    val query2 = "ALTER TABLE proyecto3 ADD IF NOT EXISTS PARTITION (hour='104700') LOCATION '/your/location/proyecto3/date=20200803/hour=104700'"
    sql(s"$query2")