如何从猪脚本中运行Mapreduce

Question

I want to understand how to integrate calling a mapreduce job from within a pig script.

I referred to the link https://wiki.apache.org/pig/NativeMapReduce

But I am not sure how to do it as how it will understand which is my mapper or reducer code. The explanation is not very clear.

If someone can illustrate it with an example, it will be of great Help.

Thanks in Advance, Cheers :)

Answer 1

Example from the pig documentation

A = LOAD 'WordcountInput.txt';
B = MAPREDUCE 'wordcount.jar' STORE A INTO 'inputDir' LOAD 'outputDir' 
    AS (word:chararray, count: int) `org.myorg.WordCount inputDir outputDir`;

In the above example, pig will store input data from A into inputDir and load the job's output data from outputDir .

Also, there is a jar in HDFS called wordcount.jar in which there is a class called org.myorg.WordCount with a main class which takes care of setting mappers and reducers, input and output etc.

You could also call the mapreduce job via hadoop jar mymr.jar org.myorg.WordCount inputDir outputDir .

Answer 2

By default pig will anticipate the map/reduce program. However hadoop comes with default mapper/reducer implementations; which is used by Pig - when map reduce class is not identified.

Further Pig uses the properties from Hadoop along with its specific properties for this. Try setting, below properties in pig script, it should be picked by Pig as well.

SET mapred.mapper.class="<fully qualified classname for mapper>"
SET mapred.reducer.class="<fully qualified classname for reducer>"

The same can be set using -Dmapred.mapper.class option as well. Comprehensive list is here Based on your hadoop installation, the properties could be as well:

mapreduce.map.class
mapreduce.reduce.class

Just FYI...

hadoop.mapred has been deprecated. Versions before 0.20.1 used mapred. Versions after that use mapreduce.

Moreover pig has its own set of properties, which can be viewed using command pig -help properties

e.g. in my pig installation, below are the properties:

The following properties are supported:
    Logging:
        verbose=true|false; default is false. This property is the same as -v switch
        brief=true|false; default is false. This property is the same as -b switch
        debug=OFF|ERROR|WARN|INFO|DEBUG; default is INFO. This property is the same as -d switch
        aggregate.warning=true|false; default is true. If true, prints count of warnings
            of each type rather than logging each warning.
    Performance tuning:
        pig.cachedbag.memusage=<mem fraction>; default is 0.2 (20% of all memory).
            Note that this memory is shared across all large bags used by the application.
        pig.skewedjoin.reduce.memusagea=<mem fraction>; default is 0.3 (30% of all memory).
            Specifies the fraction of heap available for the reducer to perform the join.
        pig.exec.nocombiner=true|false; default is false.
            Only disable combiner as a temporary workaround for problems.
        opt.multiquery=true|false; multiquery is on by default.
            Only disable multiquery as a temporary workaround for problems.
        opt.fetch=true|false; fetch is on by default.
            Scripts containing Filter, Foreach, Limit, Stream, and Union can be dumped without MR jobs.
        pig.tmpfilecompression=true|false; compression is off by default.
            Determines whether output of intermediate jobs is compressed.
        pig.tmpfilecompression.codec=lzo|gzip; default is gzip.
            Used in conjunction with pig.tmpfilecompression. Defines compression type.
        pig.noSplitCombination=true|false. Split combination is on by default.
            Determines if multiple small files are combined into a single map.
        pig.exec.mapPartAgg=true|false. Default is false.
            Determines if partial aggregation is done within map phase,
            before records are sent to combiner.
        pig.exec.mapPartAgg.minReduction=<min aggregation factor>. Default is 10.
            If the in-map partial aggregation does not reduce the output num records
            by this factor, it gets disabled.
    Miscellaneous:
        exectype=mapreduce|local; default is mapreduce. This property is the same as -x switch
        pig.additional.jars.uris=<comma seperated list of jars>. Used in place of register command.
        udf.import.list=<comma seperated list of imports>. Used to avoid package names in UDF.
        stop.on.failure=true|false; default is false. Set to true to terminate on the first error.
        pig.datetime.default.tz=<UTC time offset>. e.g. +08:00. Default is the default timezone of the host.
            Determines the timezone used to handle datetime datatype and UDFs. Additionally, any Hadoop property can be specified.