使用java.io.File.mkdirs（）将File写入目录，然后访问它在Spark Cluster模式下不起作用

Question

When i try to run the same code in client mode, the code runs successfully. But when run on cluster mode it fails to create the file and prompts with error No such File or Directory.

Below is the code sample:

new File("UnexistingLocation").mkdirs()

---> Directories created in client mode

---> Code do not give error in cluster mode but i cannot see directory created. Also while creating File inside the directory gives error No such file or Directory.

Is there a workaround by which i could create files on driver node local filesystem?

Answer 1

due to your error analysis I assume this is run in driver-scoped code. If you submit using --deploy-mode cluster your driver will be started on an arbitrary node which means that's where your directory will be. It won't be on the node where you do your spark-submit from