Add conf file to classpath in Microsoft Azure

Question

This question is similar to my other question but this time for Microsoft Azure.

We have a config file that needs to be in the classpath of the driver (and possibly the executors).

When defining a Databricks cluster in Microsoft Azure, I configured custom Spark configs: spark.executor.extraClassPath and spark.driver.extraClassPath. I pointed both paths to a directory in dbfs that contains our config. But as half suspected, this doesn't work.

The question is, which type of paths are supported by these Spark Configs? If it's only local filesystem, how do I get my config file on the driver (and the executors)?

Answer 1

Please note the setting is case sensitive and I think you should set it to a local directory, and use an init script to copy it there from dbfs (using /dbfs/path syntax), so that it's accessible with low latency from the JVM.