当我运行Sort MR作业时，压缩不起作用

Question

I configure the hadoop-2.4.0 cluster to compress the map output and my mapred-site.xml is setted like this

<property>
  <name>mapreduce.map.output.compress</name>
  <value>true</value>
</property>
<property>
  <name>mapreduce.map.output.compress.codec</name>
  <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

When I run InvertedIndex, compression works, and I can see different Counters in "Reduce shuffle bytes" .

However, when I run Sort, there is no different in "Reduce shuffle bytes" between with compression and without compression. This means compression doesn't work.

It's so weird. I get stuck. Do you know what am I doing wrong?

Answer 1

I think that if one of these programs has explicitly set in the Driver class (main method) the property: conf.setCompressMapOutput(false); , then this setting overrides the mapred-site.xml setting.