MapReduce开启压缩算法对结果进行压缩

[infobox title=”方法①:在代码中进行压缩设置”]

map阶段中添加

Configuration configuration = new Configuration();
configuration.set("mapreduce.map.output.compress","true");
configuration.set("mapreduce.map.output.compress.codec","org.apache.hadoop.io.compress.SnappyCodec");

reduce阶段中添加

configuration.set("mapreduce.output.fileoutputformat.compress","true");
configuration.set("mapreduce.output.fileoutputformat.compress.type","RECORD");
configuration.set("mapreduce.output.fileoutputformat.compress.codec","org.apache.hadoop.io.compress.SnappyCodec"); [/infobox]

[infobox title=”方法②:全局MapReduce压缩配置”]

修改配置文件mapred-site.xml

[warningbox title=”map输出数据进行压缩”]

<property>
          <name>mapreduce.map.output.compress</name>
          <value>true</value>
</property>
<property>
         <name>mapreduce.map.output.compress.codec</name>
         <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

[/warningbox]

[warningbox title=”reduce输出数据进行压缩”]

<property>       
    <name>mapreduce.output.fileoutputformat.compress</name>
     <value>true</value>
</property>
<property>         
     <name>mapreduce.output.fileoutputformat.compress.type</name>
     <value>RECORD</value>
</property>
 <property>        
      <name>mapreduce.output.fileoutputformat.compress.codec</name>
      <value>org.apache.hadoop.io.compress.SnappyCodec</value> 
</property>

[/warningbox]

修改完毕后重启集群

[/infobox]

[dangerbox title=”hadoop支持的压缩算法”]

[/dangerbox]