在python Spark中组合2个RDD

Question

I have 2 RDDs. Assume rdd1 = {'a','b','c', 'a', 'c', 'a'} and rdd2 is an output of KMeans with cluster assignment as follows -> rdd2={0,0,1,1,1,0}. I want to eventually find out how many a's and b's are there in cluster 0 and 1. For example 0 has 2 a's so something like {0, a, 2} etc. Is there a way I combine these 2 RDDS to do such an operation?

Thanks for your help.

Answer 1

The below works. using tuples and list instead of set wherever appropriate.

rdd1 = sc.parallelize(['a','b','c', 'a', 'c', 'a'])
rdd2 = sc.parallelize([0, 0, 1, 1, 1, 0])
rdd = rdd1.zip(rdd2).map(lambda x: (x,1)).reduceByKey(lambda x,y: x+y).map(lambda ((x1,x2),y): (x1,x2,y))
rdd.collect()

Output:

[('a', 0, 2), ('b', 0, 1), ('c', 1, 2), ('a', 1, 1)]