如何使用一系列键搜索ac＃dictionary？

Question

I have a dictionary with data resembling this (dictionary will be have about 100k entries):

[1] -> 5
[7] -> 50
[30] -> 3
[1000] -> 1
[100000] -> 35

I also have a list of ranges (about 1000)

MyRanges
    Range
        LowerBoundInclusive -> 0
        UpperBoundExclusive -> 10
        Total
    Range
        LowerBoundInclusive -> 10
        UpperBoundExclusive -> 50
        Total
    Range
        LowerBoundInclusive -> 100
        UpperBoundExclusive -> 1000
        Total
    Range
        LowerBoundInclusive -> 1000
        UpperBoundExclusive -> 10000
        Total
    Range (the "other" range)
        LowerBoundInclusive -> null
        UpperBoundExclusive -> null
        Total

I need to calculate the total present in the dictionary for these ranges. For example, the range 0-10 would be 55. These ranges can get really large, so I know it doesn't make sense to just search the dictionary for every value between the two ranges. My hunch is that I should get a list of keys from the dictionary, sort it, then loop through my ranges and do some sort of search to find all the keys within the ranges. Is this the correct way to do this? Is there an easy way to do that?

edit: Thanks for the responses. Real clever stuff. I forgot one pretty important caveat though. There is not the guarantee that the ranges are continuous, and the final range is everything not in the other ranges.

Answer 1

You could do something like that:

// Associate each value with the range of its key
var lookup = dictionary.ToLookup(
    kvp => ranges.FirstOrDefault(r => r.LowerBoundInclusive <= kvp.Key
                              && r.UpperBoundExclusive > kvp.Key),
    kvp => kvp.Value);

// Compute the total of values for each range
foreach (var r in ranges)
{
    r.Total = lookup[r].Sum();
}

(note: this solution doesn't take your edit into account; it doesn't handle non-contiguous ranges and the "others" range)

However, it's not very efficient if you have many ranges, since they are enumerated for each entry in the dictionary... You could get better results if you sort the dictionary by key first.

Here's a possible implementation:

// We're going to need finer control over the enumeration than foreach,
// so we manipulate the enumerator directly instead.
using (var dictEnumerator = dictionary.OrderBy(e => e.Key).GetEnumerator())
{
    // No point in going any further if the dictionary is empty
    if (dictEnumerator.MoveNext())
    {
        long othersTotal = 0; // total for items that don't fall in any range

        // The ranges need to be in ascending order
        // We want the "others" range at the end
        foreach (var range in ranges.OrderBy(r => r.LowerBoundInclusive ?? int.MaxValue))
        {
            if (range.LowerBoundInclusive == null && range.UpperBoundExclusive == null)
            {
                // this is the "others" range: use the precalculated total
                // of previous items that didn't fall in any other range
                range.Total = othersTotal;
            }
            else
            {
                range.Total = 0;
            }

            int lower = range.LowerBoundInclusive ?? int.MinValue;
            int upper = range.UpperBoundExclusive ?? int.MaxValue;

            bool endOfDict = false;
            var entry = dictEnumerator.Current;


            // keys that are below the current range don't belong to any range
            // (or they would have been included in the previous range)
            while (!endOfDict && entry.Key < lower)
            {
                othersTotal += entry.Value;
                endOfDict = !dictEnumerator.MoveNext();
                if (!endOfDict)
                    entry = dictEnumerator.Current;
            }

            // while the key in the the range, we keep adding the values
            while (!endOfDict  && lower <= entry.Key && upper > entry.Key)
            {
                range.Total += entry.Value;
                endOfDict = !dictEnumerator.MoveNext();
                if (!endOfDict)
                    entry = dictEnumerator.Current;
            }

            if (endOfDict) // No more entries in the dictionary, no need to go further
                break;

            // the value of the current entry is now outside the range,
            // so carry on to the next range
        }
    }
}

(updated to take your edit into account; works with non-contiguous ranges, and adds items that don't fall in any range to the "others" range)

I didn't run any benchmark, but it's probably pretty fast, since the dictionary and the ranges are enumerated only once.

Obviously, if the ranges are already sorted you don't need the OrderBy on ranges .

Answer 2

Consider using sorted List<T> and its BinarySearch method. If you have many queries, then each of them can be answered with O(logn) , giving total O(qlogn) time complexity, where n is the number of entries and q number of queries:

//sorted List<int> data

foreach (var range in ranges)                             // O(q)
{
    int lowerBoundIndex = data.BinarySearch(range.Start); // O(logn)
    lowerIndex = lowerIndex < 0
        ? ~lowerIndex
        : lowerIndex;

    int upperBoundIndex = data.BinarySearch(range.End);   // O(logn)
    upperBoundIndex = upperBoundIndex < 0
        ? ~upperBoundIndex - 1
        : upperBoundIndex;

    var count = (upperBoundIndex >= lowerBoundIndex)
        ? (upperBoundIndex - lowerBoundIndex + 1)
        : 0;

    // print/store count for range
}

For the dictionary case, the complexity is on average O(q*l) where q is number of queries (as above) and l is average length of the queried range. So the sorted list approach will be better if ranges are large.

Anyway, for 100k entries you should use a database, as suggested by pswg in the comments.

Answer 3

You are absolutely right, the dictionary is not the right data structure for the task.

Your idea about what to do is also right. You can improve it with some preprocessing to get the execution time to (N + Q) * Log N , where N is the number of items in the original dictionary, and Q is the number of queries that you need to run.

Here is the idea: get the items from your dictionary into a flat list, and sort it. Then preprocess the list by storing the running total in the corresponding node. Your list would end up looking like this:

• | 0 | 0 -> 0 (implicit sentinel value)
• | 1 | 1 -> 5 -- 5
• | 7 | 7 -> 55 -- 50 + 5
• | 30 | 30 -> 58 -- 3 + 50 + 5
• | 1000 | 1000 -> 59 -- 1 + 3 + 50 + 5
• | 100000 | 100000 -> 94 -- 35 + 1 + 3 + 50 + 5

With the preprocessed list in hand you can run two binary searches on the first list (ie {1, 7, 30, 1000, 100000} list) for the two ends of the query, take the totals at the current point if there was an exact match or at the point before if there wasn't an exact match, subtract the sum at the upper point from the sum at the lower point, and use that as the answer to your query.

For example, if you see the query {0, 10} you process it like this:

• Binary search on 0, get the sentinel value of 0
• Binary search on 10, get the value of 55 for 7 (no exact match on 10)
• Subtract 0 from 55 for an answer of 55.

For a query 11, 1000 you do this:

• Search 11, get 7 with the value of 55
• Search 1000, get 1000 with the value of 59
• Subtract 59-55=4 for the answer to the query.

Answer 4

The low-tech approach might be the better approach here. I'm going to make a possibly invalid assumption that your dictionary doesn't change very often; basically that queries are much more frequent than dictionary or range modifications. So you can create and cache a list of the dictionary's keys, refreshing it as required if the dictionary is modified. So, given:

List<KeyType> keys = dict.Keys.OrderBy(k => k).ToList();
List<RangeType> ranges = rangeList.OrderBy(r => r.LowerBound).ToList();

var iKey = 0;
var iRange = 0;
var count = 0;
// do a merge
while (iKey < keys.Count && iRange < ranges.Count)
{
    if (keys[iKey] < ranges[i].LowerBound)
    {
        // key is smaller than current range's lower bound
        // move to next key

        // here you could add this key to the list of keys not found in any range
        ++iKey;
    }
    else if (keys[iKey] > ranges[i].UpperBound)
    {
        // key is larger than current range's upper bound
        // move to next range
        ++iRange;
    }
    else
    {
        // key is within this range
        ++count;
        // add key to list of keys in this range
        ++iKey;
    }
}
// If there are leftover keys, then add them to the list of keys not found in a range
while (iKey < keys.Count)
{
    notFoundKeys.Add(keys[iKey]);
    ++iKey;
}

Note that this assumes non-overlapping ranges.

This algorithm is O(n), where n is the number of keys in the dictionary.

That might seem expensive, but we're only talking 100,000 comparisons, which is going to be very fast on modern hardware. The beauty of this approach is that it's dead simple to implement and it could very well be fast enough for your purposes. It's worth trying. If it's too slow then you can look at optimization.

An obvious optimization is to binary search the lower and upper bounds to get the indexes of items that fit the range. That algorithm's complexity is O(q log n), where q is the number of queries. log2(100000) is approximately 16.6. It takes two binary searches per query, so looking for 1,000 ranges will require about 33,200 key comparisons--one-third as many as with the sequential algorithm I present above.

That algorithm would look something like:

foreach (var range in ranges)
{
    int firstIndex = keys.BinarySearch(range.LowerBound);

    // See explanation below
    if (firstIndex < 0) firstIndex = ~firstIndex;

    int lastIndex = keys.BinarySearch(range.UpperBound);
    if (lastIndex < 0) lastIndex = ~lastIndex-1;

    if (keys[firstIndex] >= range.LowerBound && keys[lastIndex] <= range.UpperBound)
        count += 1 + (lastIndex - firstIndex);
}

List.BinarySearch returns the bitwise complement of the index where the next larger element would be. The code above adjusts the indexes returned if the item isn't found, to get the items that are within range.

Adding the keys not found to a list will involve keeping track of the last key found for each range, and adding that key and everything up to the first key found for the next range to the list of not found keys. It's a fairly simple modification of the code above.

A possible optimization to this algorithm would be to use the BinarySearch overload that lets you specify the starting index. After all, if you've already determined that the range 0-50 ends at index 27, there's no use searching below 27 for the range 51-100. That simple optimization could negate the advantage of the sequential search that I discuss below.

Although algorithm analysis says that this should be faster, it doesn't take into account the overhead involved in setting up each binary search, or the non-sequential memory access that can be a performance killer due to cache misses. My experiments comparing binary search to sequential search in C# (using List<T>.BinarySearch ) show that sequential search is faster when the list size is less than 10 items, although that depends somewhat on how expensive key comparisons are. On average, though, I found binary search overhead to be cost me 5 to 10 key comparisons. You have to take that into account when you're considering which algorithm would be faster.

If the number of ranges is small, the binary search algorithm will be the clear winner. But it becomes more expensive as the number of ranges grows. At some point, the sequential search algorithm, whose running time is nearly constant regardless of the number of ranges, will be faster than the binary search algorithm. Where that point is, exactly, is unclear. We know that it's something less than 3,000 ranges because n/(2*log2(n)) is equal to 3,012.

Again, since you're talking relatively small numbers, either algorithm will likely perform quite well for you. If you're hitting this thing hundreds or thousands of times per second, then you'll want to do a detailed analysis and time execution with representative data and varying numbers of ranges. If you're hitting it infrequently, then just put in something that works and worry about optimization if it becomes a performance problem.