如何有效地排序一百万个元素？

Question

I need to compare about 60.000 with a list of 935.000 elements and if they match I need to perform a calculation.

I already implemented everything needed but the process takes about 40 min. I have a unique 7-digit number in both lists. The 935.000 and the 60.000 files are unsorted. Is it more efficient to sort (which sort?) the big list before I try to find the element? Keep in mind that I have to do this calculation only once a month so I don't need to repeat the process every day.

Basically which is faster:

• unsorted linear search
• sort list first and then search with another algorithm

Answer 1

Try it out.

You've got Collections.sort() which will do the heavy lifting for you, and Collections.binarySearch() which will allow you to find the elements in the sorted list.

Answer 2

When you search the unsorted list, you have to look through half the elements on average before you find the one you're looking for. When you do that 60,000 times on a list of 935,000 elements, that works out to about

935,000 * 1/2 * 60,000 = 28,050,000,000 operations

If you sort the list first (using mergesort) it will take about n * log(n) operations. Then you can use binary search to find elements in log(n) lookups for each of the 60,000 elements in your shorted list. That's about

935,000 * log(935,000) + log(935,000) * 60,000 = 19,735,434 operations

It should be a lot faster if you sort the list first, then use a search algorithm that takes advantage of the sorted list.

Answer 3

What would work quite well is to sort both lists and then iterate over both at the same time.

Use collections.sort() to sort the lists.

You start with an index for each sorted list and just basically walk straight through it. You start with the first element on the short list and compare it to the first elements of the long list. If you reach an element on the long list with an higher 7 digit number than the current number in the short list, increment your index of the short list. This way there is no need to check elements twice.

But actually, since you want to find the intersection of two lists, you might be better off just using longList.retainAll(shortList) to just get the intersection of the two lists. Then you can perform whatever you want on both of the lists in about O(1) since there is no need to actually find anything.

Answer 4

You can sort both lists and compare them element by element incrementing first or second index ( i and j in the example below) as needed:

List<Comparable> first = ....
List<Comparable> second = ...
Collections.sort(first);
Collections.sort(second);

int i = 0;
int j = 0;
while (i < first.size() && j < second.size()) {
    if (first.get(i).compareTo(second.get(j)) == 0) {
        // Action for equals
    }
    if (first.get(i).compareTo(second.get(j)) > 0) {
        j++;
    } else {
        i++;
    }
}

The complexity of this code is O(n log(n)) where n is the biggest list size.