在一组字符串中查找超级字符串

Question

I have a list of strings, like:

cargo
cargo pants
cargo pants men buy
cargo pants men
cargo pants men melbourne buy

In this, the string that contains all remaining strings is cargo pants men melbourne buy . I'd like to remove all the shorter strings and preserve only the longest "super string".

Note, if 2 queries cargo pants and cargo shorts exist, they will be treated as 2 different queries and won't be combined.

So far, I've been doing this the brute force way - pick a string from set and walk through the same set deleting all other strings that are "substrings" of the current string. Roughly,

for (String p: big_set) {
    for (String q: big_set) {
        if (!p.equals(q)) {
            if (has_all_words(p, q)) { /* If all words in 'p' is also in 'q' */
                big_set.remove(p);
                break;
            }
        }
    }
}

Is there an intelligent algorithm to do this in less than O(n^2) time? In this function, has_all_words will preserve the order of words while comparing.

For the curious, I have a massive list of a few billion search queries (like the ones send to Google/Yahoo/Bing) and I'm trying to find hypernyms for these queries. There's a server that parses this string and produces various interesting categories. I am trying to compress the queries list in the hopes of minimizing compute cost and bandwidth. This method surely reduces bandwidth significantly (because humans can't just think of buy cargo pants melbourne in one go), but the pre-computation cost is prohibitive. And so I've been hunting for algorithms that can do this, but I haven't come across anything that does this yet.

Answer 1

• I think all you want to ask for is to remove all those sub strings which can be found in a super string .Like in the case for ["foo bar", "foo baz"] you will have to store both the strings .

• If my guess is right then yes you can achieve it in less than O(n^2). before starting with anything short each super-strings alphabetically so that no such case remains like cargo pants pants cargo men buy

• first, sort your string in decreasing order according to there
  lengths. Then pick up sub strings of the longest string (as we are
  iterating from first index and have sorted in reverse order) and
  start searching for it in rest of the strings.

• If string is found remove it and Once searching and removing completes just iterate again with the next sub string of the same super-string with the last sub-string included.

• In the end you will be left with only strings which are unique (if you consider ["foo bar", "foo baz"] as a unique string.