如何将列表简化为最不常见的字符串？

Question

I have a HashSet<string> which I'm loading vulgar words into for filtering purposes. The problem is that my list will contain "Fu" and also the word spelled out completely. What I want to do is filter the list down so it only contains "Fu", which would eliminate any other forms of the word from the list.

In other words, I want to remove all strings in the list where its substring is also a list item.

How should I go about doing this?

I have the following where excludedWords is the original HashSet , but it's not working completely:

HashSet<string> copy = new HashSet<string>(exludedWords);

foreach (string w in copy)
{
    foreach (string s in copy)
    {
        if (w.Contains(s) && w.Length > s.Length)
        {
            result.Remove(w);
        }
    }
}

Answer 1

You should compare every word in the set to every other (distinctly different) word in the set. You can accomplish this as follows (although I'm sure this is not the most efficient method, by any means):

string[] strings = { "a", "aa", "aaa", "b", "bb", "bbb", "c", "cc", "ccc" };
List<string> results = new List<string>(strings);

foreach (string str1 in strings) {
  foreach (string str2 in strings) {
    if (str1 != str2) {
      if (str2.Contains(str1)) {
        results.Remove(str2);
      }
    }
  }
}

return results;

Answer 2

Here's one way...

filter.RemoveAll(a => filter.Any(b => b != a && a.Contains(b)));

Where filter is a List and pre-populated with the filter strings.

Edit: Didn't see that you wanted Contains instead of starts with. so made the necessary mod.

Answer 3

Assuming you just want to throw away the longer values you could just use an IEqualityComparer<string> implementation to get the new set.

private class ShortestSubStringComparer : IComparer<string>, IEqualityComparer<string>
{
    public int Compare(string x, string y)
    {
        if (x == null) return (y == null) ? 0 : -1;
        if (y == null) return 1;

        Debug.Assert(x != null && y != null);
        if (this.Equals(x, y)) return x.Length.CompareTo(y.Length);
        return StringComparer.CurrentCulture.Compare(x, y);
    }

    public bool Equals(string x, string y)
    {
        if (x == null) return y == null;
        if (x.StartsWith(y)) return true;
        if (y != null && y.StartsWith(x)) return true;
        return false;
    }

    public int GetHashCode(string obj)
    {
        return obj.GetHashCode();
    }
}

And then your function can use the GroupBy function to group and select the first ordered item like so:

public HashSet<string> FindShortestSubString(HashSet<string> set)
{
    var comparer = new ShortestSubStringComparer();
    return new HashSet<string>(set.GroupBy(e => e, comparer).Select(g => g.OrderBy(e => e, comparer).First()));
}

Or possibly Min might do the trick (meaning you don't need the IComparer<string> implementation either)...

public HashSet<string> FindShortestSubString(HashSet<string> set)
{
    var comparer = new ShortestSubStringComparer();
    return new HashSet<string>(set.GroupBy(e => e, comparer).Select(g => g.Min(e => e)));
}

Answer 4

I would advise against this type of filtering. You may save some cpu cycles but you'll get some unintended consequences that may really confuse your users (or just make them plain mad)

For example, let's assume that this is you list of vulgar words...

foo bar foohead foolery

You want to filter out all of these words from some content. To be efficient you remove foohead and foolery and just filter on the substring foo.

You're going to filter innocuous words that contain foo but weren't in your orignal vulgar list.

reminds me of this recent Daily WTF... (second one down)

http://thedailywtf.com/Articles/Progree-of-enail-Status.aspx

Answer 5

You could use Regular Expressions. This is in vb but I'm sure you can convert it.

Example:

Imports System.Text.RegularExpressions
Public Class Form1

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        Dim InputString As String
        InputString = Regex.Replace(WHAT THE USER HAS ENTERED, "fu", "**")
    End Sub
End Class