Oozie java action logger logs are not shown on Oozie console

Question

I am executing a map-reduce code by calling Driver class in Oozie java action. Map reduce run successfully and I get the output as expected. However, the log statements in my driver class are not shown on oozie job logs. I am using log4j for logging in my driver class. Do I need to make some configuration changes to see the logs ?. Snippet of my workflow.xml `

<action name="MyAppDriver">
       <java>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
             <prepare>
                <delete path="${nameNode}/home/hadoop/work/surjan/outpath/20160430" />
            </prepare>
            <main-class>com.surjan.driver.MyAppMainDriver</main-class>
            <arg>/home/hadoop/work/surjan/PoC/wf-app-dir/MyApp.xml</arg>
            <job-xml>/home/hadoop/work/surjan/PoC/wf-app-dir/AppSegmenter.xml</job-xml>
        </java>
        <ok to="sendEmailSuccess"/>
        <error to="sendEmailKill"/>
    </action>

`

Answer 1

The logs going into Yarn.

In my case I have a custom java action. If you look in Yarn UI you have to dig into the mapper task that the java action is in. So in my case the oozie wf item was 0070083-200420161025476-oozie-xxx-W and oozie job -log ${wf_id} shows the java action 0070083-200420161025476-oozie-xxx-W@java1 failed with a Java exception. I cannot see any context to that. Looking on the oozie web UI only the "Job Error Log" is populated as per what is shown on the commandline. The actual logging isn't shown. The ooize job -info ${wf_id} status shows failed:

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0070083-200420161025476-oozie-xxx-W@:start:                                  OK        -                      OK         -
------------------------------------------------------------------------------------------------------------------------------------
0070083-200420161025476-oozie-xxx-W@java1                                    ERROR     job_1587370152078_1090 FAILED/KILLEDJA018
------------------------------------------------------------------------------------------------------------------------------------
0070083-200420161025476-oozie-xxx-W@fail                                     OK        -                      OK         E0729
------------------------------------------------------------------------------------------------------------------------------------

You can search for the actual yarn on the web console Yarn Resource Manager UI ( not within the "yarn logs" web console which are yarns own logs not the logs of what it is hosting). You can easily grep for the correct id on the commandline by looking for the ooize wf job id:

user@host:~/apidemo$ yarn application --list -appStates FINISHED | grep 0070083-200420161025476
20/04/22 20:42:12 INFO client.AHSProxy: Connecting to Application History server at your.host.com/130.178.58.221:10200
20/04/22 20:42:12 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
20/04/22 20:42:12 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
application_1587370152078_1090  oozie:launcher:T=java:W=java-whatever-sql-task-wf:A=java1:ID=0070083-200420161025476-oozie-xxx-W             MAPREDUCE    kerberos-id           default                FINISHED               SUCCEEDED                 100% https://your.host.com:8090/jobhistory/job/job_1587370152078_1090
user@host:~/apidemo$

Note that oozie says things failed. Yet the status of the action is "FINISHED" and the yarn application status is "SUCCESS" which seems strange.

Helpfully that commandline output also shows the url to the job history. That opens the webpage that takes you to the parent application that ran your java. If you click on the little logs link in the page you see some logs. If you look closely that page said it ran 1 operation of "task type map". If you click on the link in that row it takes you to the actual task which in my case is task_1587370152078_1090_m_000000 . You have to click into that to see the first attempt which is attempt_1587370152078_1090_m_000000_0 then on the right hand side you have a tiny logs link which shows some more specific logging.

You can also ask yarn for the logs once you know the application id:

yarn logs -applicationId application_1587370152078_1090

That showed me very detailed logs including the custom java logging that I didn't see on the console easily where I could see what was really going on.

Note that if you are writing custom code you want to let yarn set the log4j properties file rather than supply your own version so that the yarn tools can find your logs. The code will be run with a flag:

    -Dlog4j.configuration=container-log4j.properties 

The detailed logs show all the jars that are added to the classpath. You should ensure that your custom code uses the same jars and log4j version.