還原未運行，但作業已成功完成

Question

首先，我是Hadoop MapReduce的新手。 我的減速器未運行，但顯示作業已成功完成。 下面是我的控制台輸出：

• • INFO mapreduce.Job：正在運行的作業：job_1418240815217_0015
• • INFO mapreduce.Job：以超級模式運行的Job job_1418240815217_0015：false
• • INFO mapreduce.Job：地圖0％減少0％
• • INFO mapreduce.Job：地圖100％減少0％
• • INFO mapreduce.Job：作業job_1418240815217_0015成功完成
• • INFO mapreduce.Job：計數器：30

主要課程是：

public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    @SuppressWarnings("deprecation")
    Job job = new Job(conf,"NPhase2");

    job.setJarByClass(NPhase2.class);

    job.setMapOutputKeyClass(IntWritable.class);
    job.setMapOutputValueClass(NPhase2Value.class);
    job.setOutputKeyClass(NullWritable.class);
    job.setOutputValueClass(Text.class);        

    job.setMapperClass(MapClass.class);        
    job.setReducerClass(Reduce.class);

    int numberOfPartition = 0;  
    List<String> other_args = new ArrayList<String>();

    for(int i = 0; i < args.length; ++i) 
    {
        try {
            if ("-m".equals(args[i])) {
                //conf.setNumMapTasks(Integer.parseInt(args[++i]));
                ++i;
            } else if ("-r".equals(args[i])) {
                job.setNumReduceTasks(Integer.parseInt(args[++i]));
            } else if ("-k".equals(args[i])) {
                int knn = Integer.parseInt(args[++i]);
                conf.setInt("knn", knn);
                System.out.println(knn);
            } else {
                other_args.add(args[i]);
            }
            job.setNumReduceTasks(numberOfPartition * numberOfPartition);
            //conf.setNumReduceTasks(1);
        } catch (NumberFormatException except) {
            System.out.println("ERROR: Integer expected instead of " + args[i]);
        } catch (ArrayIndexOutOfBoundsException except) {
            System.out.println("ERROR: Required parameter missing from " + args[i-1]);
        }
    } 

    // Make sure there are exactly 2 parameters left.
    if (other_args.size() != 2) {
        System.out.println("ERROR: Wrong number of parameters: " +
            other_args.size() + " instead of 2.");
    }

    FileInputFormat.setInputPaths(job, other_args.get(0));
    FileOutputFormat.setOutputPath(job, new Path(other_args.get(1)));

    System.exit(job.waitForCompletion(true) ? 0 : 1);
}

我的映射器是：

公共靜態類MapClass擴展了Mapper {

    public void map(LongWritable key, Text value, Context context)  throws IOException, InterruptedException 
    {
        String line = value.toString();
        String[] parts = line.split("\\s+");
        // key format <rid1>
        IntWritable mapKey = new IntWritable(Integer.valueOf(parts[0]));
        // value format <rid2, dist>
        NPhase2Value np2v = new NPhase2Value(Integer.valueOf(parts[1]), Float.valueOf(parts[2]));
        context.write(mapKey, np2v);
    }
}

我的減速器類是：

public static class Reduce extends Reducer<IntWritable, NPhase2Value, NullWritable, Text> 
    {
        int numberOfPartition;  
        int knn;

        class Record 
    {
        public int id2;
        public float dist;

        Record(int id2, float dist) 
        {
            this.id2 = id2;
            this.dist = dist;
        }

        public String toString() 
        {
            return Integer.toString(id2) + " " + Float.toString(dist);  
        } 
    }

    class RecordComparator implements Comparator<Record> 
    {
        public int compare(Record o1, Record o2) 
        {
            int ret = 0;
            float dist = o1.dist - o2.dist;

            if (Math.abs(dist) < 1E-6)
                ret = o1.id2 - o2.id2;
            else if (dist > 0)
                ret = 1;
            else 
                ret = -1;

            return -ret;
        }   
    }

    public void setup(Context context) 
    {
        Configuration conf = new Configuration();
        conf = context.getConfiguration();
        numberOfPartition = conf.getInt("numberOfPartition", 2);    
        knn = conf.getInt("knn", 3);
    }   

    public void reduce(IntWritable key, Iterator<NPhase2Value> values, Context context) throws IOException, InterruptedException 
    {
        //initialize the pq
        RecordComparator rc = new RecordComparator();
        PriorityQueue<Record> pq = new PriorityQueue<Record>(knn + 1, rc);

        // For each record we have a reduce task
        // value format <rid1, rid2, dist>
        while (values.hasNext()) 
        {
            NPhase2Value np2v = values.next();

            int id2 = np2v.getFirst().get();
            float dist = np2v.getSecond().get();
            Record record = new Record(id2, dist);
            pq.add(record);
            if (pq.size() > knn)
                pq.poll();
        }

        while(pq.size() > 0) 
        {
            context.write(NullWritable.get(), new Text(key.toString() + " " + pq.poll().toString()));
            //break; // only ouput the first record
        }

    } // reduce
}

這是我的助手課：

公共類NPhase2Value實現WritableComparable {

private IntWritable first;
private FloatWritable second;

public NPhase2Value() {
    set(new IntWritable(), new FloatWritable());
}

public NPhase2Value(int first, float second) {
    set(new IntWritable(first), new FloatWritable(second));
}

public void set(IntWritable first, FloatWritable second) {
    this.first = first;
    this.second = second;   
}

public IntWritable getFirst() {
    return first;
}

public FloatWritable getSecond() {
    return second;
}

@Override
public void write(DataOutput out) throws IOException {
    first.write(out);
    second.write(out);
}

@Override
public void readFields(DataInput in) throws IOException {
    first.readFields(in);
    second.readFields(in);
}

@Override
public boolean equals(Object o) {
    if (o instanceof NPhase2Value) {
        NPhase2Value np2v = (NPhase2Value) o;
        return first.equals(np2v.first) && second.equals(np2v.second);
    }
    return false;
}

@Override
public String toString() {
    return first.toString() + " " + second.toString();
}

@Override
public int compareTo(NPhase2Value np2v) {
    return 1;
}

}

我使用的命令行命令是：

hadoop jar knn.jar NPhase2 -m 1 -r 3 -k 4 phase1out phase2out

我正在努力找出錯誤，但仍然無法提出解決方案。 由於時間緊迫，請在這方面幫助我。

Answer 1

因為您已將reducer任務的數量設置為0。請參見：

 int numberOfPartition = 0;  
 //.......
 job.setNumReduceTasks(numberOfPartition * numberOfPartition);

我看不到您已在代碼中的任何地方重置了numberOfPartition 。 我認為您應該在解析-r選項的地方設置它，或者完全像上面那樣完全刪除對setNumReduceTasks方法的調用，因為您在解析-r選項時已經設置了它。