地圖輸出記錄與約簡輸入記錄之間有什么關系

Question

我有這個hadoop程序：

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.hadoop.mapreduce.Job;

public class Question1_1 {

    public static class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            for (String word : value.toString().split("\\s+")) {
                context.write(new Text(word), new IntWritable(1));
            }
        }
    }

    public static class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable value : values) {
                sum += value.get();
            }
            context.write(key, new IntWritable(sum));
        }
    }



    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
        Job job = Job.getInstance(conf, "Question1_1");


        job.setJarByClass(Question1_1.class);
        job.setMapperClass(WordCountMapper.class);    
        job.setReducerClass(WordCountReducer.class);

        // Types of Key/Value
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        // Input & Output Files
        FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
        FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));


        System.exit(job.waitForCompletion(true) ? 0 : 1);

    }

}

我針對txt文件運行該程序。 我在日志中得到了這個：

    15:09:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15:09:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15:09:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15:09:12 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
15:09:12 INFO input.FileInputFormat: Total input paths to process : 1
15:09:12 INFO mapreduce.JobSubmitter: number of splits:1
15:09:12 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
15:09:12 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
15:09:12 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
15:09:12 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
15:09:12 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
15:09:12 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
15:09:12 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
15:09:12 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
15:09:12 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
15:09:12 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
15:09:12 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
15:09:12 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15:09:12 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
15:09:12 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
15:09:12 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
15:09:12 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
15:09:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local958404083_0001
15:09:12 WARN conf.Configuration: file:/tmp/hadoop-hamza/mapred/staging/hamza958404083/.staging/job_local958404083_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
15:09:12 WARN conf.Configuration: file:/tmp/hadoop-hamza/mapred/staging/hamza958404083/.staging/job_local958404083_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
15:09:12 WARN conf.Configuration: file:/tmp/hadoop-hamza/mapred/local/localRunner/hamza/job_local958404083_0001/job_local958404083_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
15:09:12 WARN conf.Configuration: file:/tmp/hadoop-hamza/mapred/local/localRunner/hamza/job_local958404083_0001/job_local958404083_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
15:09:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
15:09:12 INFO mapreduce.Job: Running job: job_local958404083_0001
15:09:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
15:09:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
15:09:12 INFO mapred.LocalJobRunner: Waiting for map tasks
15:09:12 INFO mapred.LocalJobRunner: Starting task: attempt_local958404083_0001_m_000000_0
15:09:12 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
15:09:12 INFO mapred.MapTask: Processing split: file:/home/hamza/workspace/TPIntroHadoop/flickrSample.txt:0+53568
15:09:12 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15:09:12 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15:09:12 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15:09:12 INFO mapred.MapTask: soft limit at 83886080
15:09:12 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15:09:12 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15:09:12 INFO mapred.LocalJobRunner: 
15:09:12 INFO mapred.MapTask: Starting flush of map output
15:09:12 INFO mapred.MapTask: Spilling map output
15:09:12 INFO mapred.MapTask: bufstart = 0; bufend = 62647; bufvoid = 104857600
15:09:12 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26205172(104820688); length = 9225/6553600
15:09:12 INFO mapred.MapTask: Finished spill 0
15:09:12 INFO mapred.Task: Task:attempt_local958404083_0001_m_000000_0 is done. And is in the process of committing
15:09:12 INFO mapred.LocalJobRunner: map
15:09:12 INFO mapred.Task: Task 'attempt_local958404083_0001_m_000000_0' done.
15:09:12 INFO mapred.LocalJobRunner: Finishing task: attempt_local958404083_0001_m_000000_0
15:09:12 INFO mapred.LocalJobRunner: Map task executor complete.
15:09:12 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
15:09:12 INFO mapred.Merger: Merging 1 sorted segments
15:09:12 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 41944 bytes
15:09:12 INFO mapred.LocalJobRunner: 
15:09:12 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
15:09:12 INFO mapred.Task: Task:attempt_local958404083_0001_r_000000_0 is done. And is in the process of committing
15:09:12 INFO mapred.LocalJobRunner: 
15:09:12 INFO mapred.Task: Task attempt_local958404083_0001_r_000000_0 is allowed to commit now
15:09:12 INFO output.FileOutputCommitter: Saved output of task 'attempt_local958404083_0001_r_000000_0' to file:/home/hamza/workspace/TPIntroHadoop/sresult/_temporary/0/task_local958404083_0001_r_000000
15:09:12 INFO mapred.LocalJobRunner: reduce > reduce
15:09:12 INFO mapred.Task: Task 'attempt_local958404083_0001_r_000000_0' done.
15:09:13 INFO mapreduce.Job: Job job_local958404083_0001 running in uber mode : false
15:09:13 INFO mapreduce.Job:  map 100% reduce 100%
15:09:13 INFO mapreduce.Job: Job job_local958404083_0001 completed successfully
15:09:13 INFO mapreduce.Job: Counters: 27
    File System Counters
        FILE: Number of bytes read=149581
        FILE: Number of bytes written=496089
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=100
        Map output records=2307
        Map output bytes=62647
        Map output materialized bytes=42089
        Input split bytes=122
        Combine input records=2307
        Combine output records=1218
        Reduce input groups=1218
        Reduce shuffle bytes=0
        Reduce input records=1218
        Reduce output records=1218
        Spilled Records=2436
        Shuffled Maps =0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=0
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
        Total committed heap usage (bytes)=460324864
    File Input Format Counters 
        Bytes Read=53568
    File Output Format Counters 
        Bytes Written=37479

問題： Map輸出記錄和Reduce輸入記錄之間有什么關系？ 減少輸入組代表什么？

Answer 1

在映射器中，您使用context.write(key, value)

在化Iterable<?> values器中，將來自映射器的特定key所有值組合為Iterable<?> values

為了將某些內容寫入new Path(output) ，您需要再次使用化簡器中的context

Answer 2

因此，您有以下計數器：

Map input records=100
Map output records=2307
Combine input records=2307
Combine output records=1218
Reduce input groups=1218
Reduce input records=1218
Reduce output records=1218

由此您可以確定要處理的數據：

1. 您的mapper已讀取100條輸入記錄，並生成2307鍵/值輸出記錄。 這告訴您要擴展多少數據。
2. 您有一個合並器（在這種情況下，是您的減速器），該2307已在2307投入使用，並從2307本地減少到1218 。 實際上，由於您只有一個輸入拆分（此1個映射器），所以reducer不應有任何進一步的效果。
3. 組合器輸出1218條記錄，該記錄與輸入到減速器的記錄數相匹配。
4. 精簡器接收1218條記錄，並且由於此時數據已完全精簡，因此它還告訴您有1218組。 組實際上是鍵的數量。 通常，您可以使用group和record計數來確定所使用的鍵與它們具有的值數量之間的關系。 在這種情況下，您看到組合器將數據減少了約50％，因此可以說每個鍵有大約2個值。
5. 數據已完全縮減，因此縮減器不需要做很多工作，因為此時每個鍵都有一個值，因此只輸出1218條記錄。

以我的經驗，一旦您在更大的數據集上運行並且可能正在運行映射器，則合並器的計數就變得不那么有意義了。 它們可以多次讀取和寫入相同的數據，因此您可以有效地查看映射器輸出和化簡器輸入的內容。 但是在這種情況下，您可以看到發生了什么。