如何在Hadoop流中使用opt解析器作为映射器指定python脚本

Question

How do I specify a python script which as opt parser (and accepts multiple arguments ) as a mapper in Hadoop Streaming ??

For example ,

$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
    -input myInputDirs \
    -output myOutputDir \
    -mapper myPythonScript.py \
    -reducer /bin/wc \
    -file myPythonScript.py

This the normal command to execute python script using hadoop streaming. How to mention if the myPythonScript.py has opt parser. For eg.

python myPythonscript.py -g --inputfile=Inputfilename --output=Ouputfilename -r

How do I specify this as mapper ??

Answer 1

$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
    -input myInputDirs \
    -output myOutputDir \
    -mapper "python myPythonscript.py -g --inputfile=Inputfilename --output=Ouputfilename -r" \
    -reducer /bin/wc \
    -file myPythonScript.py