Map Reduce 使用 Python 读取文本文件

Question

I'm trying to write a MapReduce program that can read an input file.But I don't really know how to use it in a MapReduce program in python.

Can someone give me a code snippet of it? I have tried the following Code to read a file in python. I have already pushed the file on HDFS file system before reading.

f = open('/usr/total.txt',"r")
g_total = int(f.readline())

I have flight data set on which i have answer the total flight per as a percentage. what i have done so far is mapper is producing key value pair <year,1> and then the reducer is aggregating the output from mapper to produce <year,total_count> .

Now I am confused how to calculate it as percentage so for that I need total number of rows.

How could I perform these all in mapper and reducer?

Answer 1

calculate [...] percentage so for that i need total number of rows

Calculating totals or averages isn't really a good use case for mapreduce. You need to force all data to one reducer with a common key, then you can sum, count, and divide totals to get percentages. For example, the mapper would output <null,year> , then the reducer would sum all yearly counts first (use a Counter object, for example), then you can sum all years together to get total number of records, since all data is available now to one reducer. Once you have that, divide by each yearly count to output <year,yearly_count/total_count> , giving a percent for each year.

As far as HDFS is concerned, you can use MrJob Python module to do this, which will run a Hadoop Streaming Job.

Worth pointing out that Pyspark or Hive can easily do the same, so you don't "need" to write MapReduce, but this is how those work behind the scenes