Spark：在大型数据集中搜索匹配项

Question

I need to count how many values of one of the columns of df1 are present in one of the columns of df2. (I just need the number of matched values)

I wouldn't be asking this question if efficiency wasn't such a big concern:

df1 contains 100,000,000+ records

df2 contains 1,000,000,000+ records

Answer 1

Just an off the top of my head idea for the case that intersection won't cut it:

For the datatype that is contained in the columns, find two hash functions h1 , h2 such that

• h1 produces hashes roughly uniformly between 0 and N
• h2 produces hashes roughly uniformly between 0 and M

such that M * N is approximately 1B, eg M = 10k , N = 100k ,

then:

• map each entry x from the column from df1 to (h1(x), x)
• map each entry x from the column from df2 to (h1(x), x)
• group both by h1 into buckets with x s
• join on h1 (that's gonna be the nasty shuffle)

then locally, for each pair of buckets (b1, b2) that came from df1 and df2 and had the same h1 hash code, do essentially the same:

• compute h2 for all b s from b1 and from b2 ,
• group by the hash code h2
• Compare the small sub-sub-buckets that remain by converting everything toSet and computing the intersection directly.

Everything that remains after intersection is present in both df1 and df2 , so compute size and sum the results across all partitions.

The idea is to select N small enough so that the buckets with M entries still comfortably fit on a single node, but at the same time prevent that the whole application dies on the first shuffle trying to find out where is what by sending every key to everyone else. For example, using SHA-256 as "hash code" for h1 wouldn't help much, because the keys would be essentially unique, so that you could take the original data directly and try to do a shuffle with that. However, if you restrict N to some reasonably small number, eg 10k, you obtain a rough approximation of where what is, so that you can then regroup the buckets and start the second stage with h2 .

Essentially it's just a random guess, I didn't test it. It could well be that the built-in intersection is smarter than everything I could possibly come up with.