从 delta lake 手动删除数据文件

Question

I have manually deleted a data file from delta lake and now the below command is giving error

mydf = spark.read.format('delta').load('/mnt/path/data')
display(mydf)

Error

A file referenced in the transaction log cannot be found. This occurs when data has been manually deleted from the file system rather than using the table `DELETE` statement. For more information, see https://docs.microsoft.com/azure/databricks/delta/delta-intro#frequently-asked-questions

i have tried restarting the cluster with no luck also tried the below

spark.conf.set("spark.sql.files.ignoreCorruptFiles", "true")
spark.conf.set("spark.databricks.io.cache.enabled", "false")

Any help on repairing the transaction log or fix the error

Answer 1

The above error indicates that you have manually deleted a data file without using the proper DELETE Statement.

As per MS Doc , you can try vacuum command. Using the vacuum command fix the error.

%sql
vacuum 'Your_path'

For more information refer this link

Answer 2

as explained before you must use vacuum to remove files as manually deleting files does not lead to the delta transaction log being updated which is what spark uses to identify what files to read.

In your case you can also use the FSCK REPAIR TABLE command. as per the docs: "Removes the file entries from the transaction log of a Delta table that can no longer be found in the underlying file system. This can happen when these files have been manually deleted."

Answer 3

FSCK Command worked for me. Thanks All