Spark Databricks 本地文件 API

Question

I'm trying to build a summary/report of the processing done in spark DataBricks. I came across below piece of code that allows the data to be written to dbfs as well as ADLS(through mount point), but the issue arises when I package the code in jar and try to execute it as a DataBricks job, I get file not found exception and now wondering how to write data into storage with out using notebooks .

    import java.io.File
    import java.io.PrintWriter

    val writer = new PrintWriter(new File("/dbfs/mnt/data/out-01/test-01"))
    writer.write("Hello Developer, Welcome to Programming.")
    writer.write("Hello Developer, Welcome to Programming 2.")
    writer.close()

I came across DButils from DataBricks, but haven't seen any sample code / documentation that I can use. Any help on it will be appreciated.

Answer 1

If your notebook created by the below of the figure below to mount ADLS, yes that you can directly write date to dbfs in the current session of your databricks.

So I think the necessary code of DBFS mount points is missing in your code which be packaged to jar file.

And please refer to the offical documents as below to see how to access directly ADLS v1 and v2 in your code.

1. Azure Data Lake Storage Gen1
2. Azure Data Lake Storage Gen2

Meanwhile, if without databricks library, you also can change your code using ADLS SDK and REST APIs to write code without using DBFS and run it in databricks.