通过蜂巢错误定制地图减少

Question

I was trying to run custom map-reduce through hive. I created sample mapper and reducer classes for wordcount. I followed below steps from this article http://www.lichun.cc/blog/2012/06/wordcount-mapreduce-example-using-hive-on-local-and-emr/

create external table if not exists raw_lines(line string)
    ROW FORMAT DELIMITED
    stored as textfile
    location '/user/new_user/hive_mr_input';

I have added sample lines for wordcount to /user/new_user/hive_mr_input dir.

create external table if not exists word_count(word string, count int)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
lines terminated by '\n' 
STORED AS TEXTFILE LOCATION '/user/new_user/hive_mr_output';

hive>
add file /home/new_user/hive/WordCountReducer.java;
add file /home/new_user/hive/WordCountMapper.java;

    from (
            from raw_lines
            map raw_lines.line        
            using '/user/new_user/hive/WordCountMapper.java'
            as word, count
            cluster by word) map_output
    insert overwrite table word_count
    reduce map_output.word, map_output.count
    using '/user/new_user/hive/WordCountReducer.java'
    as word,count;

when I executed above command I got an error:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.

I thought It maybe because of "\\t" delimiter i used in table creation so I made some changes in Mapper class and tried using file with commas

String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line,",");

and changed table structure to use "," in word_count table creation -> FIELDS TERMINATED BY ','
however I got the same error.

What's wrong with above code?

Answer 1

The reason it's now working is that your trying to use Java. In the example you point to, the author is using python. See the documentation here .

The script you supply as a custom transformation must be executable and must be able to accept input from standard input and output data from standard output. For this reason, you can actually use any language, even bash. Other popular choices are python, like in the article you linked, or ruby etc.

Whatever language you choose, you have to make sure that the interpreter is available in all the nodes, as well as all the needed libraries, otherwise the script will fail.

You're supplying java source code, and that won't work. Hive won't compile your code for you.

You can use java, but you have to build an executable jar. See this other post on how to do that .