std :: unordered_map：多线程插入？

Question

I have a bunch of data (a giant list of integers between 0 and ULLONG_MAX) and I want to extract all unique values. My approach is to create an unordered_map, using the integer list values as the keys and a throwaway bool for the map values. I iterate the list and insert throwaway values for each key. At the end I iterate the map to get all unique keys. Pretty straight forward.

However, my list is so large (100s of millions) that I'd like to multithread this process. I know a naive approach to threading won't work, because unordered_map insertions affect the underlying data structure and so it's not thread safe. And adding locks around each and every insertion will be slow and possibly negate any threading speedup.

However, presumably not every insertion changes the data structure (only ones that can't fit in the existing allocated buckets?). Is there a way to check if an unordered_map re-allocation would be required for a particular insertion, prior to inserting? That way I could only lock threads when the map is getting changed, instead of locking during every insert. Then, prior to each insertion, the threads merely check if a lock exists...rather than doing a full lock/unlock. Is that possible?

Answer 1

The fundamental rule of parallelization break the job up, work on the pieces, and then combine the pieces.

Hashing/item lookup is the most expensive part of the whole shebang, so that's what we'll focus on parallelizing.

If you absolutely need the result as a hash table, I got some bad news for you: you'll have to write your own. That being said, let's begin.

First, let's solve the problem in serial. this is simple. The below function takes a vector and a callback. We're going to take the vector, convert it into an unordered_set , and give the unordered_set to the callback. Simple? Yes.

Now, because we're going to be doing this on a thread, we can't do it right away. Instead, we'll return a lambda that takes no arguments. When that lambda is invoked, that's when it'll create the unordered_set and give it to the callback. This way, we can give each lambda to it's own thread, and each thread will run the job by invoking the lambda.

template<class Vector, class Callback>
auto lazyGetUnique(Vector& vector, Callback callback) {
    using Iterator = decltype(vector.begin());
    auto begin = vector.begin();
    auto end = vector.end();
    using elem_t = typename std::iterator_traits<Iterator>::value_type;

    //We capture begin, end, and callback
    return [begin, end, callback]() {
        callback(std::unordered_set<elem_t>(begin, end));
    };
}

Now - what should this callback do? The answer is simple: the callback should assign the contents of the unordered_set to a vector. Why? Because we're gonna be merging the results, and it's a lot faster to merge vectors than it is to merge unordered_set .

Let's write a function to give us the callback:

template<class Vector>
auto assignTo(Vector& v) {
    return [&](auto&& contents) {
        v.assign(contents.begin(), contents.end());
    };
}

Suppose we want to get the unique elements of a vector, and assign them back to that vector. This is now really simple to do:

std::vector<int> v = /* stuff */;
auto new_thread = std::thread( lazyGetUnique(v, assignTo(v)) ); 

In this example, when new_thread finishes executing, v will contain only unique elements.

Let's look at the complete function to do everything.

template<class Iterator>
auto getUnique(Iterator begin, Iterator end) {
    using elem_t = typename std::iterator_traits<Iterator>::value_type;

    std::vector<elem_t> blocks[4];

    //Split things up into blocks based on the last 4 bits
    //Of the number. This allows us to guarantee that no two blocks
    //share numbers. 
    for(; begin != end; ++begin) {
        auto val = *begin; 
        blocks[val & 0x3].push_back(val); 
    }

    //Each thread will run their portion of the problem.
    //Once it's found all unique elements, it'll stick the result in the block used as input
    auto thread_0 = std::thread( lazyGetUnique(blocks[0], assignTo(blocks[0])) );
    auto thread_1 = std::thread( lazyGetUnique(blocks[1], assignTo(blocks[1])) );
    auto thread_2 = std::thread( lazyGetUnique(blocks[2], assignTo(blocks[2])) );

    //We are thread_3, so we can just invoke it directly
    lazyGetUnique(blocks[3], assignTo(blocks[3]))(); //Here, we invoke it immediately

    //Join the other threads
    thread_0.join();
    thread_1.join();
    thread_2.join(); 

    std::vector<elem_t> result;
    result.reserve(blocks[0].size() + blocks[1].size() + blocks[2].size() + blocks[3].size());

    for(int i = 0; i < 4; ++i) {
        result.insert(result.end(), blocks[i].begin(), blocks[i].end());
    }

    return result;
}

This function breaks stuff up into 4 blocks, each of which are disjoint. It finds the unique elements in each of the 4 blocks, then combines the result. The output is a vector.