使用 Azure Databricks 从 ADLS 访问数据

Question

I am trying to access data files stored in ADLS location via Azure Databricks using storage account access keys. To access data files, I am using python notebook in azure databricks and below command works fine,

spark.conf.set(
  "fs.azure.account.key.<storage-account-name>.dfs.core.windows.net",
  "<access-key>"
)

However, when I try to list the directory using below command, it throws an error

dbutils.fs.ls("abfss://<container-name>@<storage-account-name>.dfs.core.windows.net")

ERROR:

ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, GET, https://<storage-account-name>.dfs.core.windows.net/<container-name>?upn=false&resource=filesystem&maxResults=500&timeout=90&recursive=false, AuthorizationPermissionMismatch, "This request is not authorized to perform this operation using this permission. RequestId:<request-id> Time:2021-08-03T08:53:28.0215432Z"

I am not sure on what permission would it require and how can I proceed with it.

Also, I am using ADLS Gen2 and Azure Databricks(Trial - premium).

Thanks in advance!

Answer 1

The complete config key is called "spark.hadoop.fs.azure.account.key.adlsiqdigital.dfs.core.windows.net"

However it would be beneficial for a production environment to use a service account and a mount point. This way the actions on the storage can be traced back to this application more easily than just with the generic access key and the mount point avoid specifying the connection string everywhere in your code.

Answer 2

Try this out.

spark.conf.set("fs.azure.account.key.<your-storage-account-name>.blob.core.windows.net","<your-storage-account-access-key>")
dbutils.fs.mount(source = "abfss://<container-name>@<your-storage-account-name>.dfs.core.windows.net/", mount_point = "/mnt/test")

Answer 3

You can mount ADLS storage account using access key via Databricks and then read/write data. Please try below code:

dbutils.fs.mount(
  source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net",
  mount_point = "/mnt/<mount-name>",
  extra_configs = {"fs.azure.account.key.<storage-account-name>.blob.core.windows.net":dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")})

dbutils.fs.ls("/mnt/<mount-name>")