將 dataframe 保存為 csv 文件（在 databricks 中處理）並將其上傳到 azure datalake blob 存儲

Question

I had a csv file stored in azure datalake storage which i imported in databricks by mounting the datalake account in my databricks cluster, After doing preProcessing i wanted to store the csv back in the same datalakegen2 (blobstorage) account.Any leads and help on the問題表示贊賞。謝謝。

Answer 1

只需在相同的安裝位置寫入文件。 請參閱此處的示例： https://docs.databricks.com/spark/latest/data-sources/azure/azure-datalake-gen2.html#example-notebook

df.write.json("abfss://<file_system>@<storage-account-name>.dfs.core.windows.net/iot_devices.json")

Answer 2

只需將其直接保存到 Blob 存儲即可。

df.write.
    format("com.databricks.spark.csv").
    option("header", "true").
    save("myfile.csv")

在本地保存文件然后將其推送到 Blob 中沒有意義。