从集群获取现有的mapreduce作业（该作业可能正在运行或已完成）

Question

Previously, I was using org.apache.hadoop.mapred.JobClient#getJob(org.apache.hadoop.mapred.JobID) to get the RunningJob . This call was made from the job completion callback method, however, seems to me that there is a timing issue where if the job is already completed then the above getJob() method cannot find it and returns null. I can confirm that the job was completed from the cluster UI.

Keeping the RunningJob apart, is there a way to get the org.apache.hadoop.mapreduce.Job object of a mapred job given the org.apache.hadoop.mapreduce.JobID , regardless whether the job is currently running or is completed?

I tried to code up something like:

Cluster cluster = jobClient.getClusterHandle(); Job job = cluster.getJob(JobID.forName(jobId)); log.info("Trying to get actual job with id {} , found {} on cluster {}", JobID.forName(jobId), job, cluster);

I can see the right jobId, and can also see the cluster object.. but the cluster.getJob() method returns null, so the job itself is null.

Is there something that I'm missing out here?

Answer 1

The problem was with with a recent yarn upgrade that required enabling MR history server on my system. This fixed the issue. I recently upgraded from MR v1 to v2 and in that upgrade, all completed jobs are now moved to the history server.

Answer 2

You look for getAllJobStatuses() that return JobStatus[] :

  List<JobStatus> runningJobs = new ArrayList<JobStatus>();
  List<JobStatus> completedJobs = new ArrayList<JobStatus>();
  for (JobStatus job : cluster.getAllJobStatuses()) {
    if (!job.isJobComplete()) {
      runningJobs.add(job);
    }
    else {
      completedJobs.add(job)
    }
  }

  // list of running JobIDs
  for (JobStatus rjob : runningJobs) {
        System.out.println(rjob.getJobID().toString());
  }
  // list of completed JobIDs
  for (JobStatus cjob : completedJobs) {
        System.out.println(cjob.getJobID().toString());
  }

  // to print out short report on running jobs:
  // displayJobList(runningJobs.toArray(new JobStatus[0]));