Hadoop流式处理-files python出现中文乱码

Question

I am not sure "garbled" is the correct word for my issue. My issue is like this. I use hadoop-streaming-0.20.2-cdh3u6.jar and python to write a mapreduce. The command is like below:

hadoop jarhadoop-streaming-0.20.2-cdh3u6.jar \
 -D mapred.reduce.tasks=0 \
 -D mapred.job.priority=VERY_HIGH \
 -files probability,rule_dict,dict \
 -file "naive_bayes.py" \
 -mapper "python naive_bayes.py" \
 -input xxx \
 -output xxx

the probability,rule_dict,dict are directories in my local storage, files in the them contained Chinese words. when I use python to read these files I get bad "garbled".A little fragment of files:

 石块 0.000025
 最后 0.000321
 老面孔 0.000012
 GP17 0.000012
 圈 0.000136
 看个够 0.000062
 布兰 0.000062

and what read from these files were

PB�;��W�mVKZm����e�2��U�؎��"/�1_�u�8)i~�J�N86�l}[��y8�;%(K��/P��<��F/+����=��X�n�

is there a way to solve my issue ?

I have uploaded the same python script and same directories to the mapper machine, run it from command line, and it running well without this issue.

words in files are utf-8.

Answer 1

I have found the issue's root-cause. When upload directory with -files, hadoop will create .file_name.crc file in the same directory, so when my code iteration files will also process .file_name.crc, so when encode words in these files will crash.