在带有通配符的集合中查找单词的最快方法

Question

I have ~200,000 words that I need to find matches in with words that can contain any number of letter wildcards. I also need the option to look up words without any wildcards.

I've separated the words into collections by length:

static readonly HashSet<string>[] _validWords = { 
    new HashSet<string>(StringComparer.OrdinalIgnoreCase), // 3 letter words
    new HashSet<string>(StringComparer.OrdinalIgnoreCase), // 4 letter words
    new HashSet<string>(StringComparer.OrdinalIgnoreCase), // 5 letter words
    new HashSet<string>(StringComparer.OrdinalIgnoreCase), // 6 letter words
    new HashSet<string>(StringComparer.OrdinalIgnoreCase), // 7 letter words
    new HashSet<string>(StringComparer.OrdinalIgnoreCase), // 8 letter words
    new HashSet<string>(StringComparer.OrdinalIgnoreCase)  // 9 letter words
};

To search for a specific word is simple:

public static bool IsValid(string word) {
    return word.Length >= GameplaySettings.Instance.MinWordLength && _validWords[word.Length - 3].Contains(word);
}

This is my current implementation of finding wildcard words, using a Regex (eventually I'd like to get all matching words, but for now just finding one (or none) is fine.):

public static bool IsValidRegex(string pattern, int length) {
    pattern = $"^{pattern}$";

    foreach (string word in _validWords[length - 3]) {
        if (Regex.Matches(word, pattern, RegexOptions.Singleline).Count > 0) { return true; }
    }

    return false;
}

There can be any number of wildcards (eg all letters can even be wildcards), and it's currently not performing as well as I'd hope.

So I'm wondering if there is more efficient method!

Thanks for any help/suggestions!

Answer 1

Since your wildcards can only match one letter, the problem isn't too hard. If you needed to support variable length substrings, I'd suggest you go and read some of the scientific literature on how regular expressions work.

This is a fairly basic 2nd year comp-sci "data structures and algorithms" exercise. Using a Dictionary in every Node probably isn't going to be the fastest / most memory efficient. But I would tackle the problem like this;

class Node
{
    public bool endWord;
    public Dictionary<char, Node> next;
}

public class Words
{
    private Node root = new Node { endWord = false };
    public const char wildcard = '_';

    public void DefineWord(string word)
    {
        var node = root;
        foreach (var c in word)
        {
            if (node.next == null)
                node.next = new Dictionary<char, Node>();
            if (node.next.TryGetValue(c, out var nextNode))
            {
                node = nextNode;
            }
            else
            {
                node = node.next[c] = new Node { endWord = false };
            }
        }
        node.endWord = true;
    }

    private bool IsValid(ReadOnlySpan<char> word, Node node)
    {
        if (word.IsEmpty && node.endWord)
            return true;
        if (node.next == null)
            return false;

        if (word[0] == wildcard)
        {
            word = word.Slice(1);
            foreach(var n in node.next.Values)
            {
                if (IsValid(word, n))
                    return true;
            }
        } else if (node.next.TryGetValue(word[0], out var nextNode))
            return IsValid(word.Slice(1), nextNode);
        return false;
    }

    public bool IsValid(string word)
        => IsValid(word, root);

    public static void Test1()
    {
        var words = new Words();
        words.DefineWord("APE");
        words.DefineWord("APPLE");
        words.DefineWord("BEAR");
        words.DefineWord("BEER");
        words.DefineWord("PEAR");
        words.DefineWord("PEER");
        words.DefineWord("PEERS");

        Assert.True(words.IsValid("APE"));
        Assert.True(words.IsValid("APPLE"));
        Assert.True(words.IsValid("PEAR"));
        Assert.True(words.IsValid("PEER"));
        Assert.True(words.IsValid("PEERS"));
        Assert.True(!words.IsValid("PLIERS"));
        Assert.True(words.IsValid("PE_R"));
        Assert.True(words.IsValid("_EAR"));
        Assert.True(words.IsValid("_E_R"));
    }
}