Databricks - pyspark.pandas.Dataframe.to_excel 不识别 abfss 协议

Question

I want to save a Dataframe (pyspark.pandas.Dataframe) as an Excel file on the Azure Data Lake Gen2 using Azure Databricks in Python. I've switched to the pyspark.pandas.Dataframe because it is the recommended one since Spark 3.2.

There's a method called to_excel ( here the doc) that allows to save a file to a container in ADL but I'm facing problems with the file system access protocols. From the same class I use the methods to_csv and to_parquet using abfss and I'd like to use the same for the excel.

So when I try so save it using:

import pyspark.pandas as ps
# Omit the df initialization
file_name = "abfss://CONTAINER@SERVICEACCOUNT.dfs.core.windows.net/FILE.xlsx"
sheet = "test"
df.to_excel(file_name, test)

I get the error from fsspec:

ValueError: Protocol not known: abfss

Can someone please help me?

Thanks in advance!

Answer 1

The pandas dataframe does not support the protocol. It seems on Databricks you can only access and write the file on abfss via Spark dataframe. So, the solution is to write file locally and manually move to abfss. See this answer here .

Answer 2

You can not save it directly but you can have it as its stored in temp location and move it to your directory. My code piece is:

import xlsxwriter import pandas as pd1 

workbook = xlsxwriter.Workbook('data_checks_output.xlsx') 

worksheet = workbook.add_worksheet('top_rows') 

Create a Pandas Excel writer using XlsxWriter as the engine.

writer = pd1.ExcelWriter('data_checks_output.xlsx', engine='xlsxwriter') 

output = dataset.limit(10) 
output = output.toPandas() 
output.to_excel(writer, sheet_name='top_rows',startrow=row_number)

writer.save()

After write.save

run below code, which is nothing but moves temp location of file to your desginated location.

Below code does the work of moving files.

%sh
sudo mv file_name.xlsx /dbfs/mnt/fpmount/