如何有效地检查给定的字符串是否包含数组中的单词

Question

I'm building an app where a user inputs a block of text. Once submitted I need to check if that block of text contains words from my predefined list of words.
The list of words is large, say around 50K, so I need to figure out a way I can efficiently and quickly do the check.
Here are some solutions I've thought up, but they seem really inefficient

Option 1: Creating a function in the App's code that just loops through each predefined word and checks if that word is in the block of text

eg

var wordList = ['fox','dog','tree'];  //in my app this list will be large
function contains(userInput) {
    for(i in wordList){
        if(userInput.indexOf(wordList[i]) > -1)
            return true;       
    }
    return false
}

Option 2: Both the block of text and word list will be stored in the DB, so I could do an SQL statement like this

eg

SELECT *
FROM UserInput ui
    INNER JOIN WordList wl ON wl.word LIKE CONCAT('%', ui.InputText, '%')

Is there a better way to do this?

Answer 1

If you're looking at anything bigger than a small data set (and 50k qualifies), then I'd definitely do any data manipulation in the database.

You're correct that an open-ended LIKE isn't going to be terribly performant, but it'll be orders of magnitude faster than doing it outside of a database. If your user input is guaranteed to be a full word, then you could break everything in WordList in to separate words and do an exact match search. If you're not guaranteed to have a full word from UserInput , then I'd use your option 2.

If performance is super-important, then you could also look in to full text indices

• MySQL: http://dev.mysql.com/doc/refman/5.6/en/innodb-fulltext-index.html
• MS SQL: http://msdn.microsoft.com/en-us/library/ms187317.aspx .

Answer 2

Aho–Corasick string matching algorithm

There is link on C# implementation in article.

Answer 3

You can do this either in the database or using LINQ.

Split your user input on the space so you have an Array or Table Valued Parameter containing the user input words. Then just do an inner join against your word list. Anything left after the join will be words that exist in both places. Performance will be excellent.

SELECT SomeColumn
FROM WordList wl
JOIN @tvp ui ON wl.SomeColumn = ui.SomeColumn

It will be orders of magnitude faster than doing LIKE searches, and a lot simpler to set up than full text indexing.

Answer 4

I would definitely do it on the application side.. but I'm supposing the list as a "bad words" list.. and it won't change often.. if the assumption is correct.. then the code would be something like this

static List<String> Chached;
List<String> GetBadWords()
{
    if(Chached==null)
    {
         //load words from db into static array
         Chached.Sort();//!important step
    }
    return Chached
}

public bool IstextValid(String sText)
{
    List<String> oBadWords = GetBadWords()
    foreach(String sWord in Rexex.Split(sText,@"\W"))//split by anything not alphanumeric
        if(oBadWords.BinarySearch(sWord )>=0)//since is sorted we can do binary search O(log n)
            return false;
    return true;
}

do basically there are two optimizations to consider

• keep the list in memory, no need to constantly nag sql
• use binary search to avoid O(N * M)