Mapreduce Python

Question

I'm completely new to Python and MapReduce. It would be great if someone can help me to achieve below results. I want to calculate the count of key and the average of values per key from a list like below. The first number in the pair is the key and the second is the value.

• 1,5
• 1,5
• 2,7
• 2,8
• 1,10
• 2,10
• 3,3
• 1,20

The output will look as below.

• 1, 4, 10
• 2, 3, 8.3
• 3, 1, 3

Thank you

Answer 1

I would recommend you to use itertools instead of reduce.

import itertools
import functools
import statistics

data = [[1,5], [1,5], [2,7], [2,8], [1,10], [2,10], [3,3], [1,20]]

# First, sort and group the input by key
sorted_data = sorted(data, key=lambda x: x[0])
grouped = itertools.groupby(sorted_data, lambda e: e[0])

# This will result in a structure like this:
# [
#   (1, [[1, 5], [1, 5], [1, 10], [1, 20]]),
#   (2, [[2, 7], [2, 8], [2, 10]]),
#   (3, [[3, 3]])
# ]

# Remove the duplicate keys from the structure
remove_duplicate_keys = map(lambda x: (x[0], [e[1] for e in x[1]]), grouped)

# This will produce the following structure:
# [
#   (1, [5, 5, 10, 20]),
#   (2, [7, 8, 10]),
#   (3, [3])
# ]

# Now, calculate count and mean for each entry
result = map(lambda x: (x[0], len(x[1]), statistics.mean(x[1])), remove_dublicate_keys)

# This will result in the following list:
# [(1, 4, 10), (2, 3, 8.333333333333334), (3, 1, 3)]

Note: All instructions will return generators. This means python will not calculate anything until you start using it. But you can only access the elements once. If you need them to be in a regular list or need to access the information multiple times, replace the last line with this:

result = list(map(lambda x: (x[0], len(x[1]), statistics.mean(x[1])), remove_dublicate_keys))

This will convert the original generator chain into a regular list.