PySpark ReduceByKey

Question

I have been trying to make it work for a while, but failed every time. I have 2 files. One has a list of names:

Name1
Name2
Name3
Name4

The other is list of values associated with names for each day in the year over several years:

['0.1,0.2,0.3,0.4',
 '0.5,0.6,0.7,0.8', 
 '10,1000,0.2,5000'
  ...]

The goal is to have the output like:

Name1: [0.1,0.5,10]
Name2: [0.2,0.6,1000]
Name3:[0.3,0.7,0.2]
Name4:[0.4,0.8,5000]

And then plot histogram for each. I wrote a mapper that creates a list of tuples that produces the following output (this is an RDD object):

[[('Name1', [0.1]),('Name2', [0,2]),('Name3', [0.3]),('Name4', [0.4])],
[('Name1', [0.5]),('Name2', [0,6]),('Name3', [0.7]),('Name4', [0.8])],
[('Name1', [10]),('Name2', [1000]),('Name3', [0.8]),('Name4', [5000])]]

Now I need to concatenate all values for each name in a single list, but each map by key, value that I attempted returns a wrong result.

Answer 1

You can simply loop through each and create a dictionary from it using dict.setdefault() . Example -

>>> ll = [[('Name1', [0.1]),('Name2', [0,2]),('Name3', [0.3]),('Name4', [0.4])],
... [('Name1', [0.5]),('Name2', [0,6]),('Name3', [0.7]),('Name4', [0.8])],
... [('Name1', [10]),('Name2', [1000]),('Name3', [0.8]),('Name4', [5000])]]
>>> d = {}
>>> for i in ll:
...     for tup in i:
...             d.setdefault(tup[0],[]).extend(tup[1])
...
>>> pprint.pprint(d)
{'Name1': [0.1, 0.5, 10],
 'Name2': [0, 2, 0, 6, 1000],
 'Name3': [0.3, 0.7, 0.8],
 'Name4': [0.4, 0.8, 5000]}

---

For Pyspark RDD Object, try a simple reduce function such as -

func = lambda x,y: x+y

Then send this in to reduceByKey method -

object.reduceByKey(func)

---

Per comments, actually the OP has a list of RDD Objects (not a single RDD Objects) , in that case you can convert the RDD objects to a list by calling .collect() and then do the logic , and then you can decide whether you want the resultant as a python dictionary or an RDD object, if you want first. You can call dict.items() to get the key-value pairs and call sc.parrallelize . Example -

d = {}
for i in ll:
    c = i.collect()
    for tup in i:
            d.setdefault(tup[0],[]).extend(tup[1])

rddobj = sc.parallelize(d.items())