Access Azure Data Lake Storage Gen2 using the account key

Question

I'm trying to retrieve all paths to directories in Azure Data Lake Storage Gen2 using method specified in doc: https://docs.databricks.com/external-data/azure-storage.html , in particular I'm trying to access only using the storgae account key (not via Azure service principals or SAS tokens).

The doc says the we just need to set this conf:

spark.conf.set(
    "fs.azure.account.key.<storage-account>.dfs.core.windows.net",
    dbutils.secrets.get(scope="<scope>", key="<storage-account-access-key>"))

But n.netheless I'm getting 403 permssion error:

ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, GET, https://<storage-account>.dfs.core.windows.net/<my-container>?upn=false&resource=filesystem&maxResults=5000&timeout=90&recursive=false, AuthorizationPermissionMismatch, "This request is not authorized to perform this operation using this permission. RequestId:ef2753bb-501f-00bf-3c6c-26d7f4000000 Time:2023-01-12T09:56:54.1924327Z"
    at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:248)

how come I'm still getting the error if I only use storage key? I'm a bit confusing becasue using service principal I have to grant the client_id on the storage but in this case what I have to grant? I'm just using the key..

any help is appreciated Thanks

Answer 1

I tried to reproduce the same in my environment and got the below results

Go to storage account -> IAM -> +Add Storage Blob Contributor.

Configure storage account in two ways.

with key vault

spark.conf.set( "fs.azure.account.key.<storage_account>.dfs.core.windows.net", dbutils.secrets.get(scope="<scope_name>", key="<Access_key>"))

without key vault

spark.conf.set("fs.azure.account.key.<storage_account>.dfs.core.windows.net","Access_key")

for more information refer this SO thread.