删除重复项的好程序

Question

I've a huge string(1GB) with SPACE delimiter, I'll convert it into Array[]. My string contains lots of duplicates. I've to sort the string and remove duplicates. I've made up 2 procedures and I'm not able to decide one among these two.

Procedure 1

I assume that sorting string is costly process, I wanted to remove duplicates using HashSet and then sort.

Procedure 2

I sort the Array and remove duplicates using formal procedure of comparing sorted Array with its previous value to next value and remove duplicates.

From my point of view, 1st procedure seems good. But I'm not aware if I run into any errors. Which one will be good..?

Answer 1

Assuming memory is not an issue, the most efficient approach, performance-wise, is probably:

String s = someOneGbString();
String[] words = s.split("\\s+");
Set<String> noDupes = new HashSet<>();
Collections.addAll(noDupes, words);

And if you need it sorted:

Set<String> sorted = new TreeSet<> (noDupes);

Or with Java 8:

Set<String> sorted = Arrays.stream(s.split("\\s+"))
                           .sorted()
                           .collect(toSet());

Answer 2

You can use external merge sort. http://en.wikipedia.org/wiki/External_sorting#External_merge_sort

Read the whole String. Split it into an array ( String[] ). Use in-place quicksort. Iterate over the array and check if sequential neighboring strings are the same or not. As substrings are not copies of the original String but simply refers to the memory location in the String pool, this will be space efficient.

Time Complexity: O(nlogn)

Do as others suggested. Use a TreeSet or a HashSet. For TreeSet, each insertion will be O(logn) so total is O(nlogn). However this will be less efficient than quicksort in terms of both time and space. HashSet is more complicated depending on the hash function. Under most circumstances, it will do OK, with O(n) time complexity.