无法从 Azure DataBricks [wasbs vs abfss] 在 Storage Gen2 上创建挂载

Question

I am trying to create a mount point in Azure Storage Gen2 over a container, using the syntax given on Azure docs. I found 2 ways using 'abfss' for Gen2 and 'wasbs' for regular blob storage. Since I am using 'Storage Gen2', so using 'abfss' but that not working. Although if I use 'wasb' its able to mount. Not sure why. I am confused

Syntax-1

url = "wasbs://"+container+"@"+storage_name+".blob.core.windows.net"
config  = "fs.azure.account.key."+storage_name+".blob.core.windows.net"

Syntax-2

url="abfss://"+container+"@"+storage_name+".dfs.core.windows.net"
as_config  = "fs.azure.account.key."+storage_name+".dfs.core.windows.net

When I use Syntax-2, gets error:

ExecutionError: An error occurred while calling o246.mount. : java.lang.NullPointerException: authEndpoint at shaded.databricks.v20180920_b33d810.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)

Answer 1

You can't mount the abfss using the storage key - it works only for wasbs (as it's confirmed by you). Mounting of abfss is possible only using service principal, as it's described in documentation :

configs = {"fs.azure.account.auth.type": "OAuth",
          "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
          "fs.azure.account.oauth2.client.id": "<application-id>",
          "fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope="<scope-name>",key="<service-credential-key-name>"),
          "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<directory-id>/oauth2/token"}

# Optionally, you can add <directory-name> to the source URI of your mount point.
dbutils.fs.mount(
  source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/",
  mount_point = "/mnt/<mount-name>",
  extra_configs = configs)