Spark Session Builder Cannot Find Resource File

Question

For context, I'm trying to configure my Spark session to use fair scheduling. For that reason I have a file, fairscheduler.xml, at the 'root' of my resources folder. This file is correctly packaging into my build folder in the location I'd expect.

To further prove that's working, I have this line:

val mypath = getClass.getResource("/fairscheduler.xml")

That is working and returns exactly the path I'd expect: /<some path>/<my jar>/fairscheduler.xml

The following is throwing a FileNotFound exception:

sessionBuilder
    ...
    .config("spark.scheduler.mode", "FAIR")
    .config("spark.scheduler.allocation.file", mypath.toString) <- THIS LINE
    .config("spark.scheduler.pool", "fair_pool")
    ...
    .getOrCreate

Just for sanity, I have logged out mypath.toString .

Why is it that the spark session builder can't recognize a resource file that otherwise seems valid? Does it expect "spark configuration" files to exist somewhere specific? Or am I completely missing some dumb little thing here?

Answer 1

This is not a Apache Spark limitation but an expected behaviour of how resource files are handled in Java JAR files. Question How to get a path to a resource in a Java JAR file provides more details and a workaround of persisting the content in a temp file.