Optimization of nested loops using LINQ

Question

Can you please suggest how to write an optmized LINQ query for the following operation?

foreach (DataRow entry1 in table1.Rows)
{
    var columnA = entry1["ColumnA"] as string;
    if (!string.IsNullOrEmpty(columnA))
    {
        foreach (string entry2 in table2)
        {
            var dataExists = table3.Any(rows3 =>
                !string.IsNullOrEmpty(rows3[entry2] as string)
                && columnA.IsEqual(rows3["ColumnB"] as string));
            if (dataExists)
            {
                entry1[entry2] = Compute(columnA, entry2);
            }
        }
    }
}

I tried with this, but the results don't match in terms of the unique iteration counts.

var t2t3Pair = from entry2 in table2
    let entry3 = table3.FirstOrDefault(x =>
        !string.IsNullOrEmpty(x[entry2] as string))
    where entry3 != null
    select new { entry2, entry3 };

var t1t3Pair = from pair in t2t3Pair
    from entry1 in table1.AsEnumerable()
    let columnA = entry1["ColumnA"] as string
    where !string.IsNullOrEmpty(columnA)
       && columnA.IsEqual(pair.entry3["ColumnB"] as string)
    select new { Entry1Alias = entry1, Entry2Alias = pair.entry2 };

foreach (var pair in t1t3Pair)
{
    var columnA = (string)pair.Entry1Alias["ColumnA"];
    pair.Entry1Alias[pair.Entry2Alias] = Compute(columnA, pair.Entry2Alias);
}

Note: IsEqual is my extension method to compare string without case sensitivity.

Answer 1

Apparently the bottleneck is the line

var dataExists = table3.Any(rows3 =>
    !string.IsNullOrEmpty(rows3[entry2] as string)
    && columnA.IsEqual(rows3["ColumnB"] as string));

which is executed inside the innermost loop.

As usual, it can be optimized by preparing in advance a fast lookup data structure and use it inside the critical loop.

For your case, I would suggest something like this:

var dataExistsMap = table3.AsEnumerable()
    .GroupBy(r => r["ColumnB"] as string)
    .Where(g => !string.IsNullOrEmpty(g.Key))
    .ToDictionary(g => g.Key, g => new HashSet<string>(
        table2.Where(e => g.Any(r => !string.IsNullOrEmpty(r[e] as string)))
    // Include the proper comparer if your IsEqual method is using non default string comparison
    //, StringComparer.OrdinalIgnoreCase
    )
);

foreach (DataRow entry1 in table1.Rows)
{
    var columnA = entry1["ColumnA"] as string;
    if (string.IsNullOrEmpty(columnA)) continue;
    HashSet<string> dataExistsSet;
    if (!dataExistsMap.TryGetValue(columnA, out dataExistsSet)) continue;
    foreach (string entry2 in table2.Where(dataExistsSet.Contains))
        entry1[entry2] = Compute(columnA, entry2);
}