如何有效删除Spark RDD中的子集

Question

When conducting research, I find it somewhat difficult to delete all the subsets in Spark RDD.

The data structure is RDD[(key,set)] . For example, it could be:

RDD[ ("peter",Set(1,2,3)), ("mike",Set(1,3)), ("jack",Set(5)) ]

Since the set of mike (Set(1,3)) is a subset of peter's (Set(1,2,3)) , I want to delete "mike", which will end up with

RDD[ ("peter",Set(1,2,3)), ("jack",Set(5)) ]

It is easy to implement in python locally with two "for" loop operation. But when I want to extend to cloud with scala and spark, it is not that easy to find a good solution.

Thanks

Answer 1

This can be achieved by using RDD.fold function.
In this case the output required is a "List" (ItemList) of superset items. For this the input should also be converted to "List" (RDD of ItemList)

import org.apache.spark.rdd.RDD

// type alias for convinience
type Item = Tuple2[String, Set[Int]]
type ItemList = List[Item]

// Source RDD
val lst:RDD[Item] = sc.parallelize( List( ("peter",Set(1,2,3)), ("mike",Set(1,3)), ("jack",Set(5)) ) )


// Convert each element as a List. This is needed for using fold function on RDD
// since the data-type of the parameters are the same as output parameter 
// data-type for fold function
val listOflst:RDD[ItemList] = lst.map(x => List(x))

// for each element in second ItemList
// - Check if it is not subset of any element in first ItemList and add first
// - Remove the subset of newly added elements
def combiner(first:ItemList, second:ItemList) : ItemList = {
    def helper(lst: ItemList, i:Item) : ItemList = {
        val isSubset: Boolean = lst.exists( x=> i._2.subsetOf(x._2))
        if( isSubset) lst else i :: lst.filterNot( x => x._2.subsetOf(i._2))
    }
    second.foldLeft(first)(helper)
}


listOflst.fold(List())(combiner)

Answer 2

I doubt we can escape to comparing each element to each other (the equivalent of a double loop in a non-distributed algorithm). The subset operation between sets is not reflexive, meaning that we need to compare is "alice" subsetof "bob" and is "bob" subsetof "alice" .

To do this using the Spark API, we can resort to multiplying the data with itself using a cartesian product and verifying each entry of the resulting matrix:

val data = Seq(("peter",Set(1,2,3)), ("mike",Set(1,3)), ("anne", Set(7)),("jack",Set(5,4,1)), ("lizza", Set(5,1)), ("bart", Set(5,4)), ("maggie", Set(5)))
// expected result from this dataset = peter, olga,  anne, jack
val userSet = sparkContext.parallelize(data)
val prod = userSet.cartesian(userSet)
val subsetMembers = prod.collect{case ((name1, set1), (name2,set2)) if  (name1 != name2) && (set2.subsetOf(set1)) && (set1 -- set2).nonEmpty => (name2, set2) }
val superset = userSet.subtract(subsetMembers)    

// lets see the results:
superset.collect()
// Array[(String, scala.collection.immutable.Set[Int])] = Array((olga,Set(1, 2, 3)), (peter,Set(1, 2, 3)), (anne,Set(7)), (jack,Set(5, 4, 1)))

Answer 3

You can use filter after a map.

You can build like a map that will return a value for what you want to delete. First build a function:

def filter_mike(line):
    if line[1] != Set(1,3):
        return line
    else:
        return None

Then you can filter now like this:

your_rdd.map(filter_mike).filter(lambda x: x != None)

This will work