Databricks、dbutils、获取 Azure Data Lake gen 2 路径中所有子文件夹的文件计数和文件大小

Question

I'm coding in a Databricks notebook (pyspark) and trying to get the filecount and filesizes of all subfolders in a specific Azure Data Lake gen2 mount path using dbutils.

I have code for it on a specific folder but I'm stuck on how to write the recursive part...

Answer 1

How about this?

def deep_ls(path: str):
    """List all files in base path recursively."""
    for x in dbutils.fs.ls(path):
        if x.path[-1] is not '/':
            yield x
        else:
            for y in deep_ls(x.path):
                yield y

Credits to

https://forums.databricks.com/questions/18932/listing-all-files-under-an-azure-data-lake-gen2-co.html

https://gist.github.com/Menziess/bfcbea6a309e0990e8c296ce23125059

Answer 2

Get the list of the files from directory, Print and get the count with the below code.

def get_dir_content(ls_path):
  dir_paths = dbutils.fs.ls(ls_path)
  subdir_paths = [get_dir_content(p.path) for p in dir_paths if p.isDir() and p.path != ls_path]
  flat_subdir_paths = [p for subdir in subdir_paths for p in subdir]
  return list(map(lambda p: p.path, dir_paths)) + flat_subdir_paths

paths = get_dir_content('dbfs:/')

or

paths = get_dir_content('abfss://')

Below line prints the file names with path and number of files count at the end.

len([print(p) for p in paths])

if you want only count number of files use the below:

len([p for p in paths])