如何在 DBFS 上挂载 Azure 数据湖存储

Question

I need to mount Azure Data Lake Store Gen1 data folders on Azure Databricks File System using Azure Service Principal Client credentials. Please help on the same

Answer 1

There are three ways of accessing Azure Data Lake Storage Gen1:

1. Pass your Azure Active Directory credentials, also known as credential passthrough.
2. Mount an Azure Data Lake Storage Gen1 filesystem to DBFS using a service principal and OAuth 2.0.
3. Use a service principal directly.

1. Pass your Azure Active Directory credentials, also known as credential passthrough:

You can authenticate automatically to Azure Data Lake Storage Gen1 from Azure Databricks clusters using the same Azure Active Directory (Azure AD) identity that you use to log into Azure Databricks. When you enable your cluster for Azure AD credential passthrough, commands that you run on that cluster will be able to read and write your data in Azure Data Lake Storage Gen1 without requiring you to configure service principal credentials for access to storage.

Enable Azure Data Lake Storage credential passthrough for a standard cluster

For complete setup and usage instructions, see Secure access to Azure Data Lake Storage using Azure Active Directory credential passthrough .

2. Mount an Azure Data Lake Storage Gen1 filesystem to DBFS using a service principal and OAuth 2.0.

Step1: Create and grant permissions to service principal

If your selected access method requires a service principal with adequate permissions, and you do not have one, follow these steps:

1. Create an Azure AD application and service principal that can access resources. Note the following properties:

   application-id: An ID that uniquely identifies the client application.

   directory-id: An ID that uniquely identifies the Azure AD instance.

   service-credential: A string that the application uses to prove its identity.

2. Register the service principal, granting the correct role assignment, such as Contributor, on the Azure Data Lake Storage Gen1 account.

Step2: Mount Azure Data Lake Storage Gen1 resource using a service principal and OAuth 2.0

Python code:

configs = {"<prefix>.oauth2.access.token.provider.type": "ClientCredential",
           "<prefix>.oauth2.client.id": "<application-id>",
           "<prefix>.oauth2.credential": dbutils.secrets.get(scope = "<scope-name>", key = "<key-name-for-service-credential>"),
           "<prefix>.oauth2.refresh.url": "https://login.microsoftonline.com/<directory-id>/oauth2/token"}

# Optionally, you can add <directory-name> to the source URI of your mount point.
dbutils.fs.mount(
  source = "adl://<storage-resource>.azuredatalakestore.net/<directory-name>",
  mount_point = "/mnt/<mount-name>",
  extra_configs = configs)

3. Access directly with Spark APIs using a service principal and OAuth 2.0

You can access an Azure Data Lake Storage Gen1 storage account directly (as opposed to mounting with DBFS) with OAuth 2.0 using the service principal.

Access using the DataFrame API:

To read from your Azure Data Lake Storage Gen1 account, you can configure Spark to use service credentials with the following snippet in your notebook:

spark.conf.set("<prefix>.oauth2.access.token.provider.type", "ClientCredential")
spark.conf.set("<prefix>.oauth2.client.id", "<application-id>")
spark.conf.set("<prefix>.oauth2.credential","<key-name-for-service-credential>"))
spark.conf.set("<prefix>.oauth2.refresh.url", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

Reference: Azure Databricks - Azure Data Lake Storage Gen1