How to retrieve yarn's logs programmatically using java

Question

I actually retrieve my spark application's logs on a linux shell with :

yarn logs -applicationId applicationid

Is there any way to retrieve it programmatically using java ?

Answer 1

Yes you can. You can get most of the key information about an application via YarnClient and you can make rest calls to the Spark History Server API . The endpoint you're looking for here is

/applications/[base-app-id]/logs

Answer 2

I wanted to do it programmatically using java, so i finally took a look at the code behind the command :

yarn logs -applicationId applicationid

which is in :

src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java

I now retrieve the logs in a string (content). The code is:

String applicationId = "application_1492795815045_3940";
ApplicationId appId = appId = ConverterUtils.toApplicationId(applicationId);
LogCLIHelpers logCliHelper = new LogCLIHelpers();
Configuration config = new Configuration();
logCliHelper.setConf(config);
String appOwner = UserGroupInformation.getCurrentUser().getShortUserName();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
PrintStream ps = new PrintStream(baos);
// Function to retrieve logs
logCliHelper.dumpAllContainersLogs(appId, appOwner, ps);
String content = new String(baos.toByteArray(), StandardCharsets.UTF_8);
System.out.println(content)

Answer 3

Your method in shell environment is correct!

In my opinion, because yarn is already an executable program in your system.

To make current java process(ie, current jvm) access and use it. You can start up a new child process to help you do the job.

Maybe the code followed will help you.

public class YarnLog {
    //
    public static void getYarnLog(String appid) throws IOException {
        BufferedReader br = null;
        try {
            Process p = Runtime.getRuntime().exec(String.format("yarn logs -applicationId %s", appid));
            br = new BufferedReader(new InputStreamReader(p.getInputStream()));
            String line;
            while((line = br.readLine()) != null) {
                System.out.println(line);
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if(br != null) {
                br.close();
            }
        }
    }
}

After the successful termination of this child process, you can use your specific logs as normal files in you current working directory .