Oozie java action and spark action in same workflow error

Question

I have a workflow whicha has two actions - java and spark. In the lib directory If I keep the both Jars, then the Java action completes but the spark action throws error as

2017-12-01 03:50:02,612 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.TaskLog.createLogSyncer()

Having two lib paths results to java action not triggering. How to keep java jar and spark jar in same workflow.

Answer 1

1. You should first try to resolve this issue by using the lib version that spark is using. What is happening is your java and spark actions are using a common lib but with different versions. Java is able to run as "most likely" the lib jar version that java action requires is coming before in the class path.

2. If you want to go for the dirty solution, shade your java action jar and relocate the conflicting package name. (relocate package using shaded plugin)