How to minimize linq queries and increase the performance

Question

var Table = new List();

var partlist = Table
    .AsEnumerable()
    .Where(row =>
    {
        if (!string.IsNullOrEmpty(row.Field<string>("Column1")) &&
            row.Field<string>("Column1").Equals(variableName1) &&
            !string.IsNullOrEmpty(row.Field<string>("Column2")) &&
            row.Field<string>("Column2").Equals(variableName2))
            return true;
        return false;
    })
    .Select(row =>
    {
        return new
        {
            PartNumber = row.Field<string>("HardwareType"),
            Number = row.Field<string>("HardwareSerialNo")
        };
    })
    .Distinct();

var distinctParts =
    partlist
        .Select(part => { return part.PartNumber; })
        .Distinct();

foreach (var distinctPart in distinctParts)
{
    var list = partlist.Where(part =>
    {
        if (part.PartNumber.Equals(distinctPart))
            return true;
        return false;
    })
    .Select(part => { return part.Number; })
    .Distinct();

    int quantity = list.Count();
    hwList[distinctPart] = quantity;
}

The above code is working fine but its taking very long to execute. Is there any way to minimize the code and increase the performance. Please help me out

Answer 1

The problem here is that you are not using the correct data type. For every entry in distinctParts it needs to iterate partlist . And that is a huge problem, because partlist defines a query and not the result of a query. In other words, for each iteration of the foreach loop, the partlist query is executed, executing all the code you defined for it:

1. Extracting the data from the rows
2. Creating new instances of the anonymous type
3. Removing the duplicate entries.

This is due to the deferred nature of LINQ queries.

I would go about this problem by extracting a dictionary from partlist that directly contains the data you need. This would execute the partlist query exactly once:

var parts = partlist.GroupBy(x => x.PartNumber)
                    .ToDictionary(x => x.Key,
                                  x => x.Select(y => y.Number)
                                        .Distinct().Count()));

foreach (var kvp in parts)
    hwList[kvp.Key] = kvp.Value;

Please note: I additionally removed the need for distinctParts with this.
This should be a lot faster.

Answer 2

You could abstract out some of your filtering but otherwise I don't see places to enhance:

function bool IsMatch(row, column, match)
{
    var value = row.Field<string>(column);
    return !string.IsNullOrEmpty(value) && value.Equals(match);
}

which only really makes it more readable:

.Where(row => {
    return IsMatch(row, "Column1", variableName1) && 
           IsMatch(row, "Column2", variableName2);
 })

---

similar thing goes with your anonymous object. If you create a light little class you can abstract the implementation to the constructor:

public class PartModel
{
    public PartModel(TableRow row)
    {
        PartNumber = row.Field<string>("HardwareType");
        Number = row.Field<string> ("HardwareSerialNo");
    }
    public string PartNumber { get; set; }
    public string Number { get; set; }
}

which simplifies that to:

.Select(row => { return new PartModel(row) }; })