将数据帧从 Azure Databricks 笔记本写入 Azure DataLake Gen2 表

Question

I've created a DataFrame which I would like to write / export next to my Azure DataLake Gen2 in Tables (need to create new Table for this).

In the future I will also need to update this Azure DL Gen2 Table with new DataFrames.

In Azure Databricks I've created a connection Azure Databricks -> Azure DataLake to see my my files:

Appreciate help how to write it in spark / pyspark.

Thank you!

Answer 1

Steps to write dataframe from Azure Databricks Notebook to Azure Data Lake Gen2:

Step1: Access directly using the storage account access key

Step2: Using DBUTILS to list the files in the storage account

Step3: Use the previosult established DBFS mount point to read the data and create the data frame.

Step4: Write data into Azure Data Lake Gen2 account

Read the airline csv file and write the output to parquet format for easy query

For more details, refer " Tutorial: Azure Data Lake Storage Gen2, Azure Databricks & Spark ".

Hope this helps. Do let us know if you any further queries.

Answer 2

我建议不要以 parquet 格式写入数据，而是使用 Delta 格式，它内部使用 Parquet 格式但提供其他功能，如 ACID 事务。语法是

df.write.format("delta").save(path)