高效的字符串索引以进行全文本索引

Question

I am looking for a data structure to solve the following problem. Receive as input a large collection of rather short strings (say 50 million, less than 30 characters) and index them as you like. Then, answer queries where I give a new string and you provide strings from the initial set which are similar to the string provided (say, the 10 best such strings). The notion of "similarity" would ideally be something like edit distance or Jaro-Winkler distance, or an approximation thereof, but it should be resilient to minor changes in spelling and word order, and to the addition of junk words. (For instance, unlike a standard indexing task, requesting "foo bar" should yield "foo" if it is indeed the closest string in the collection).

To give an example, suppose the string collection is {"Charles Dickens", "Mary Shelley", "Robert Stephenson"}. Querying "Dickens, Charles" should find "Charles Dickens". Querying "by Shelley" should return "Mary Shelley".

The trivial approach where you compute the similarity of the query string to all strings in the collection one by one is too slow for a large collection. What would be a good data structure to answer such queries more efficiently? Ideally, I would be looking for a good Java implementation of this.

Answer 1

Two suggestions come to mind:

1) Pick a distance function that satisfies the triangle inequality and use a http://en.wikipedia.org/wiki/Cover_tree - might provide some speedup but probably not orders of magnitude.

2) Guess that the closest match will include at least one stretch of k contiguous characters that is an exact match between the two strings. Build a data-structure that eg with hash table lookups can find all the strings in the collection that have at least k contiguous characters the same as some part of the query string, and then use your distance function to see which of the strings returned from this is the best match. Should be fast but will sometimes miss the right answer.

Answer 2

As an alternative to your trivial approach you can solve the problem in two steps:

1. Build an index of words which occurs in all string, which allows you to find sentences which contain given word. This should be much smaller than 50 million (if we are speaking about natural language). And you may not care about "foop bar"->"foo" because you have only words.
2. Split your query into words. For each word find all sentences containing this word. For each sentence compute the similarity with the query string using your metric.

The other benefit is that in many cases you can change your metric without rebuilding words index.