在合理的时间内在两个大数据集中查找公共元素

Question

I have two Spark dataframes DFa and DFb, they have same schema, ('country', 'id', 'price', 'name').

• DFa has around 610 million rows,
• DFb has 3000 milllion rows.

Now I want to find all rows from DFa and DFb that have same id, where id looks like "A6195A55-ACB4-48DD-9E57-5EAF6A056C80".

It's a SQL inner join, but when I run Spark SQL inner join, one task got killed because container used too much memory and caused Java heap memory error. And my cluster has limited resources, tuning YARN and Spark configuration is not a feasible method.

Is there any other solution to deal with this? Not using spark solution is also acceptable if the runtime is acceptable.

More generally, Can anyone give some algorithms and solutions when find common elements in two very large datasets.

Answer 1

First compute 64 bit hashes of your ids. The comparison will be a lot faster on the hashes, than on the string ids.

My basic idea is:

• Build a hash table from DFa.
• As you compute the hashes for DFb, you do a lookup in the table. If there's nothing there then drop the entry (no match). If you get a hit compare the actual IDs to make sure you don't get a false positive.

The complexity is O(N). Not knowing how many overlaps you expect this is the best you can do since you might have to output everything, because it all matches.

The naive implementation would use about 6GB of ram for the table (assuming 80% occupancy and that you use a flat hash table), but you can do better. Since we already have the hash, we only need to know if it's exists. So you only need one bit to mark that which reduces the memory usage by a lot (you need 64x less memory per entry, but you need to lower occupancy). However this is not a common datastructure so you'll need to implement it.

But there's something even better, something even more compact. That is called bloom filter. This will introduce some more false positives, but we had to double check anyway because we didn't trust the hash, so it's not a big downside. The best part is that you should be able to find libraries for it already available.

So everything together it looks like this:

• Compute hashes from DFa and build a bloom filter.
• Compute hashes from DFb and check against the bloom filter. If you get a match lookup the ID in DFa to make sure it's a real match and add it to the result.

Answer 2

This is a typical usecase in any big data environment. You can use the Map-Side joins where you cache the smaller table which is broadcasted to all the executors.

You can read more about broadcasted joins here

Broadcast-Joins