[英]Not able to delete directory from Azure Storage container by Databricks notebook
I'm trying to delete empty directories from Azure storage container which mounted to my DBFS我正在尝试从安装到我的 DBFS 的 Azure 存储容器中删除空目录
I'm able to list all directories which has no files.我能够列出所有没有文件的目录。
%sh
find /dbfs/mnt/test/logs/2021 -empty -type d
Result:结果:
/dbfs/mnt/test/logs/2021/02/12
/dbfs/mnt/test/logs/2021/02/15
/dbfs/mnt/test/logs/2021/02/16
But when I try to delete them, it is failing with Resource temporary unavailable.但是当我尝试删除它们时,由于资源暂时不可用而失败。
%sh
find /dbfs/mnt/test/logs/ -type d -exec rmdir {} \;
Result:结果:
rmdir: failed to remove '/dbfs/mnt/test/logs/': Directory not empty
rmdir: failed to remove '/dbfs/mnt/test/logs/2021': Directory not empty
rmdir: failed to remove '/dbfs/mnt/test/logs/2021/02': Directory not empty
rmdir: failed to remove '/dbfs/mnt/test/logs/2021/02/12': Resource temporarily unavailable
I'm able to successfully remove files older than certain days.. removing direcotry is not working.我能够成功删除某些天以前的文件。删除目录不起作用。 (Below command to remove files working
(以下命令删除正在工作的文件
%sh
find /dbfs/mnt/test/logs/ -name "*.log" -type f -mtime +5 -exec rm -f {} \;
First thing to remember - DBFS is an abstraction over the cloud blob storage, where there is no real directories - they are just prefixes that are used to organize data.首先要记住 - DBFS 是对云 blob 存储的抽象,其中没有真正的目录 - 它们只是用于组织数据的前缀。 And if you do
%sh ls -ls /dbfs/mnt/test/logs/
you may notice that all directories will have the same timestamp, and it could be the recent one - I don't remember out the head how it's calculated.如果您执行
%sh ls -ls /dbfs/mnt/test/logs/
您可能会注意到所有目录都将具有相同的时间戳,并且可能是最近的一个 - 我不记得它是如何计算的了。 Only files have the timestamp.只有文件有时间戳。
So if you need to reliably remove directories, it's better to use dbutils.fs.rm('/mnt/test/logs/', True)
(in Python, or similar in the Scala) to remove directory recursively (see docs ).因此,如果您需要可靠地删除目录,最好使用
dbutils.fs.rm('/mnt/test/logs/', True)
(在 Python 中,或在 Scala 中类似)递归地删除目录(参见docs )。 But there are limitations, like there is no support for wildcards, etc., so you need to generate a list of directories to delete, and do the deletion.但是有一些限制,比如不支持通配符等,所以需要生成要删除的目录列表,然后进行删除。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.