如何在 databricks/Azure 数据湖中保存 15k csv 文件

Question

I've a question how should I download a.csv files from Auzre data lake then make some calculation and save this in.csv again. I know that for downloading.csv I can use: data=pd.read_csv('example.csv') #example

new_data=data//2+data #calculation in databricks notebook and now the question is how to save new_data in.csv format in Azure Data lake with the name: example_calulated.csv

Answer 1

To access files from ADLS you need to Mount an Azure Data Lake Storage Gen2 filesystem to DBFS.

To read files from ADLS use the code below.

df = spark.read.format("csv").option("inferSchema", "true").option("header", "true").option("delimiter",",").load(file_location)

After applying transformations on data, you can write data in CSV file. Follow below code.

target_folder_path = 'path_to_adls_folder '

 
#write as CSV data

df.write.format("CSV").save("example_calulated.csv ")

Then you will have to rename saved csv file using dbutils.fs.mv

Although it rather copies and deletes the old file. There is no real rename function for Databricks

dbutils.fs.mv(old_name, new_name)

For more information you refer this article by Ryan Kennedy

To rename 15K files you can refer to this similar issue answered by sri sivani charan