从 Databricks 笔记本中的 Azure Data Lake Storage Gen1 获取嵌套文件夹的大小

Question

I want fetch folder size details from Databricks Notebook.

We can do the same via putty by running hadoop fs -lh {root-folder-path}. This command will return human readable size of the all the folder inside root-folder. PFB sample :

I tried running similar hadoop command from notebook as below but Hadoop is not installed in driver node I believe :

When I tried ls {root-folder-path}. I am getting folder size as 0. This is because, dbutils provide size value for files only. Folders are hardcoded to 0.PFB sample :

Kindly guide me the best way to fetch the details.

Answer 1

In Azure Databricks, this is expected behavior.

• For Files, it displays the actual file size.
• For Directories, it displays the size=0
• For Corrupted files displays the size=0

You can get more details using Azure Databricks CLI:

The following article " Computing total storage size of a folder in Azure Data Lake with Pyspark " explains how to recursively compute the storage size and the number of files and folders in ADLS Gen 1 into Databricks.