Oozie上的Apache Spark：是否提供额外的配置文件？

Question

I'm building a Spark app (1.6.0 currently) to run on Cloudera with Oozie in Hue. We want to use Hue and Oozie, as the people who will be running these jobs will be more comfortable with a browser-based interface than hacking around in Oozie XML configs or firing off spark-submit on the command-line.

I've figured out how to run a basic Oozie/Spark Action via Hue (3.10). But we want to be able to provide various non-Spark parameters via a config file at runtime, as you would for a normal Scala app. I'm struggling to find a mechanism that (a) Oozie will accept, and (b) Spark will recognise for picking up the config params from the specified file at runtime.

I've tried various permutations eg putting this as one of the Options in the Oozie Action "Properties" tab in Hue:

options "-Dconfig.file=/my/file/location/fubar.conf"

But the Spark job fails to pick up the config, or just fails completely (no obvious error in logs).

Running the Spark code in local mode (ie not on Cloudera) from the command-line using spark-submit seems to work:

spark-submit --class com.example.Sparky --master local[*] \
--driver-java-options "-Dconfig.file=/my/file/location/fubar.conf" \
target/scala-2.11/spark-dummy_2.11-1.0.jar

So I guess I need to find out how to supply the equivalent runtime config to an Oozie/Spark Action on Cloudera.

Anybody know the right way to do this?

Answer 1

So it turns out you can specify the options as Java driver options for the Oozie Spark Action.

You can edit the Spark Action to set various properties via the little cog symbol in the top right corner of the initial page of the form.

Click on the cogs to open the second page of the form, then select the "Properties" tab.

In "Options list", enter the same Java driver options as in the spark-submit example:

--driver-java-options "-Dconfig.file=/my/file/location/fubar.conf"

So this allows you to pass properties into your Spark app that might otherwise be set via your app config file. For example, if you have a property "app.fubar.var1", you can pass this in directly via the Java driver options.

--driver-java-options "-Dapp.fubar.var1=myvalue"

But I still cannot get my Spark app to recognise the location of my config file if I pass it into the Oozie Spark Action like this.