如何使用R列出Azure Databricks上数据湖文件系统中的子目录

Question

I am working in an R-notebook in databricks on Azure. Using AzureStor package, I can list the names of objects in the data lake or the file system therein, resp., the following way:

 endPoint <- AzureStor::adls_endpoint(endpoint = "https://<myStorageName>.dfs.core.windows.net" ,key = <myStorageKey>)
 storage_containers <- AzureStor::list_storage_containers(endPoint)

 paste0("https://", myStorageName,".dfs.core.windows.net/", names(storage_containers)[1]) -> path2fs
 myFileSys <- AzureStor::adls_filesystem(path2fs, key)
 AzureStor::list_adls_files(myFileSys, "/")

That gives my an R data.frame that comprises information about the "name" of the content and also a column "isDirectory".

If "isDirectory" is true, I would like to see the content of this directory. How does that work? Trying to set a new endpoint as

 endPoint <- AzureStor::adls_endpoint(endpoint = "https://<myStorageName>.dfs.core.windows.net/<myDirectoryName>" ,key = <myStorageKey>)

fails.

So, how can I further let my code explore the directory and its content when the structure is like DataLake -> FileSystem -> Directory -> Directory&Files -> Directory&Files -> ... etc.?

Answer 1

The answer to my question is just to set recursive = TRUE, so:

 list_adls_files(myFileSys, dir = "/", info = "all", recursive = TRUE)

Can be so easy sometimes!