文件作为命令行参数在 Oozie 工作流中触发操作

Question

How to pass a file as command line argument to spark job in Oozie workflow? My spark job is expecting a file as command line argument, but when I am pass that file in the workflow as /file/location it is not picking up that file.

Answer 1

I got one workaround, if we put the file in a custom-directory in ozzie shared library with a few additional change in job.properties

• oozie.use.system.libpath=true
• oozie.action.sharelib.for.spark=spark,custom-directory
• oozie.libpath=true

Then we need to update the shared lib using below command:

oozie admin -auth SIMPLE -sharelibupdate

After that we can directly pick up the file by just using the name of the file, which we placed in custom-directory, in the oozie workflow.