当使用 SPARK 读取视图时，在 HUDI 表上创建 Athena 视图会返回软删除记录

Question

I have multiple HUDI tables with differing column names and I built a view on top of it to standardize the column names. When this view is read from Athena, it returns a correct response. But, when the same view is read using SPARK using spark.read.parquet("<>"), it returns the soft deleted records too. I understand a HUDI table needs to be read with spark.read.format("hudi") but since this is a view on it, I have to use spark.read.parquet(""). Is there a way to enforce HUDI to retain only the latest commit in the table and suppress all the old commits?

Answer 1

Athena view is a virtual table store in the metastore Glue, the best way to have the same result of Athena in Spark is by using AWS Glue as metastore/catalog for your spark session. To do that you can use this lib which allows you to use AWS Glue as an Hive metastore, then you can read the view using spark.read.table("<database name>.<view name>") or via an SQL query:

val df = spark.sql("SELECT * FROM <database name>.<view name>")

Try to avoid spark.read.parquet("") because it doesn't use the hudi metadata at all, if you have issues with Glue, you can use Hive to create the same view you created in Athena for spark.