如何加快LINQ WHERE？

Question

I have run a profiler on my .NET winforms app (compiled with .NET 4.7.1) and it is pointing at the following function as consuming 73% of my application's CPU time, which seems like far too much for a simple utility function:

public static bool DoesRecordExist(string keyColumn1, string keyColumn2, string keyColumn3,
        string keyValue1, string keyValue2, string keyValue3, DataTable dt)
{
    if (dt != null && dt.Rows.Count > 0) {
        bool exists = dt.AsEnumerable()
            .Where(r =>
                string.Equals(SafeTrim(r[keyColumn1]), keyValue1, StringComparison.CurrentCultureIgnoreCase) &&
                string.Equals(SafeTrim(r[keyColumn2]), keyValue2, StringComparison.CurrentCultureIgnoreCase) &&
                string.Equals(SafeTrim(r[keyColumn3]), keyValue3, StringComparison.CurrentCultureIgnoreCase)
            )
            .Any();
        return exists;
    } else {
        return false;
    }
}

The purpose of this function is to pass in some key column names and matching key values, and checking whether any matching record exists in the in-memory c# DataTable .

My app is processing hundreds of thousands of records and for each record, this function must be called multiple times. The app is doing a lot of inserts, and before any insert, it must check whether that record already exists in the database. I figured that an in-memory check against the DataTable would be much faster than going back to the physical database each time, so that's why I'm doing this in-memory check. Each time I do a database insert, I do a corresponding insert into the DataTable , so that subsequent checks as to whether the record exists will be accurate.

So to my question: Is there a faster approach? (I don't think I can avoid checking for record existence each and every time, else I'll end up with duplicate inserts and key violations.)

EDIT #1 In addition to trying the suggestions that have been coming in, which I'm trying now, it occurred to me that I should also maybe do the .AsEnumerable() only once and pass in the EnumerableRowCollection<DataRow> instead of the DataTable . Do you think this will help?

EDIT #2 I just did a controlled test and found that querying the database directly to see if a record already exists is dramatically slower than doing an in-memory lookup.

Answer 1

Your solution find all occurences which evaluates true in the condition and then you ask if there is any. Instead use Any directly. Replace Where with Any. It will stop processing when hits first true evaulation of the condition.

bool exists = dt.AsEnumerable().Any(r => condition);

Answer 2

You should try parallel execution, this should be a very good case for that as you mentioned you are working with a huge set, and no orderliness is needed if you just want to check if a record already exists.

bool exists = dt.AsEnumerable().AsParallel().Any((r =>
            string.Equals(SafeTrim(r[keyColumn1]), keyValue1, StringComparison.CurrentCultureIgnoreCase) &&
            string.Equals(SafeTrim(r[keyColumn2]), keyValue2, StringComparison.CurrentCultureIgnoreCase) &&
            string.Equals(SafeTrim(r[keyColumn3]), keyValue3, StringComparison.CurrentCultureIgnoreCase)
        )

Answer 3

It might be that you want to transpose your data structure. Instead of having a DataTable where each row has keyColumn1 , keyColumn2 and keyColumn3 , have 3 HashSet<string> , where the first contains all of the keyColumn1 values, etc.

Doing this should be a lot faster than iterating through each of the rows:

var hashSetColumn1 = new HashSet<string>(
    dt.Rows.Select(x => x[keyColumn1]),
   StringComparison.CurrentCultureIgnoreCase);

var hashSetColumn2 = new HashSet<string>(
    dt.Rows.Select(x => x[keyColumn2]),
   StringComparison.CurrentCultureIgnoreCase);

var hashSetColumn3 = new HashSet<string>(
    dt.Rows.Select(x => x[keyColumn3]),
   StringComparison.CurrentCultureIgnoreCase);

Obviously, create these once, and then maintain them (as you're currently maintaining your DataTable). They're expensive to create, but cheap to query.

Then:

bool exists = hashSetColumn1.Contains(keyValue1) &&
    hashSetColumn2.Contains(keyValue2) &&
    hashSetColumn3.Contains(keyValue3);

Alternatively (and more cleanly), you can define your own struct which contains values from the 3 columns, and use a single HashSet:

public struct Row : IEquatable<Row>
{
    // Convenience
    private static readonly IEqualityComparer<string> comparer = StringComparer.CurrentCultureIngoreCase;

    public string Value1 { get; }
    public string Value2 { get; }
    public string Value3 { get; }

    public Row(string value1, string value2, string value3)
    {
        Value1 = value1;
        Value2 = value2;
        Value3 = value3;
    }

    public override bool Equals(object obj) => obj is Row row && Equals(row);

    public bool Equals(Row other)
    {
        return comparer.Equals(Value1, other.Value1) &&
               comparer.Equals(Value2, other.Value2) &&
               comparer.Equals(Value3, other.Value3);
    }

    public override int GetHashCode()
    {
        unchecked
        {
            int hash = 17;
            hash = hash * 23 + comparer.GetHashCode(Value1);
            hash = hash * 23 + comparer.GetHashCode(Value2);
            hash = hash * 23 + comparer.GetHashCode(Value3);
            return hash;
        }
    }

    public static bool operator ==(Row left, Row right) => left.Equals(right);
    public static bool operator !=(Row left, Row right) => !(left == right);
}

Then you can make a:

var hashSet = new HashSet<Row>(dt.Select(x => new Row(x[keyColumn1], x[keyColumn2], x[keyColumn3]));

And cache that. Query it like:

hashSet.Contains(new Row(keyValue1, keyValue2, keyValue3));

Answer 4

I suggest that you are keeping the key columns of the existing records in a HashSet . I'm using tuples here, but you could also create your own Key struct or class by overriding GetHashCode and Equals .

private HashSet<(string, string, string)> _existingKeys =
    new HashSet<(string, string, string)>();

Then you can test the existence of a key very quickly with

if (_existingKeys.Contains((keyValue1, keyValue2, keyValue3))) {
    ...
}

Don't forget to keep this HashSet in sync with your additions and deletions. Note that tuples cannot be compared with CurrentCultureIgnoreCase . Therefore either convert all the keys to lower case, or use the custom struct approach where you can use the desired comparison method.

public readonly struct Key
{
    public Key(string key1, string key2, string key3) : this()
    {
        Key1 = key1?.Trim() ?? "";
        Key2 = key2?.Trim() ?? "";
        Key3 = key3?.Trim() ?? "";
    }

    public string Key1 { get; }
    public string Key2 { get; }
    public string Key3 { get; }

    public override bool Equals(object obj)
    {
        if (!(obj is Key)) {
            return false;
        }

        var key = (Key)obj;
        return
            String.Equals(Key1, key.Key1, StringComparison.CurrentCultureIgnoreCase) &&
            String.Equals(Key2, key.Key2, StringComparison.CurrentCultureIgnoreCase) &&
            String.Equals(Key3, key.Key3, StringComparison.CurrentCultureIgnoreCase);
    }

    public override int GetHashCode()
    {
        int hashCode = -2131266610;
        unchecked {
            hashCode = hashCode * -1521134295 + StringComparer.CurrentCultureIgnoreCase.GetHashCode(Key1);
            hashCode = hashCode * -1521134295 + StringComparer.CurrentCultureIgnoreCase.GetHashCode(Key2);
            hashCode = hashCode * -1521134295 + StringComparer.CurrentCultureIgnoreCase.GetHashCode(Key3);
        }
        return hashCode;
    }
}

Another question is whether it is a good idea to use the current culture when comparing db keys. Users with different cultures might get different results. Better explicitly specify the same culture used by the db.

Answer 5

In some cases using LINQ won't optimize as good as a sequential query, so you might be better of writing the query just the old-fashined way:

public static bool DoesRecordExist(string keyColumn1, string keyColumn2, string keyColumn3,
        string keyValue1, string keyValue2, string keyValue3, DataTable dt)
{
    if (dt != null) 
    {
        foreach (var r in dt.Rows)
        {
            if (string.Equals(SafeTrim(r[keyColumn1]), keyValue1, StringComparison.CurrentCultureIgnoreCase) &&
                string.Equals(SafeTrim(r[keyColumn2]), keyValue2, StringComparison.CurrentCultureIgnoreCase) &&
                string.Equals(SafeTrim(r[keyColumn3]), keyValue3, StringComparison.CurrentCultureIgnoreCase)
            {
                return true;
            }
        }
    }
    return false;
}

But there might be more structural improvements, but this depends on the situation whether you can use it.

Option 1: Making the selection already in the database You are using a DataTable , so there is a chance that you fetch the data from the database. If you have a lot of records, then it might make more sense to move this check to the database. When using the proper indexes it might be way faster then an in-memory tablescan.

Option 2: Replace string.Equals+SafeTrim with a custom method You are using SafeTrim up to three times per row, which creates a lot of new strings. When you create your own method that compares both strings (string.Equals) with respect to leading/trailing whitespaces (SafeTrim), but without creating a new string then this could be way faster, reduce memory load and reduce garbage collection. If the implementation is good enough to inline, then you'll gain a lot of performance.

Option 3: Check the columns in the proper order Make sure you use the proper order and specify the column that has the least probability to match as keyColumn1 . This will make the if-statement result to false sooner. If keyColumn1 matches in 80% of the cases, then you need to perform a lot more comparisons.