优化慢速LINQ查询

Question

I needed to optimize the following loop that takes 20 seconds to run:

    foreach (IGrouping<DateTime, DateTime> item in groups)
    {
        var countMatchId = initialGroups
                        .Where(grp => CalculateArg(grp.a.Arg) == item.Key && grp.b.Arg == someId)
                        .Sum(y => y.c.Value);

        var countAll = initialGroups
                        .Where(grp => CalculateArg(grp.a.Arg) == item.Key)
                        .Sum(y => y.c.Value);
    }

...where CalculateArg is a relatively expensive function. I thought, CalculateArg must be the culprit therefore should only be used in one query, so I came up with this:

    foreach (IGrouping<DateTime, DateTime> item in groups)
    {
        var result = initialGroups
                        .Where(grp => CalculateArg(grp.a.Arg) == item.Key);

        var countMatchId = result
                        .Where(x => x.c.Arg == someId).Sum(y => y.c.Value);

        var countAll = result
                        .Sum(y => y.c.Value);

The problem with this result, is that it only saves about 200milliseconds, so that didn't optimize anything. I still have for countMatchId the .Where() that iterates all elements, and the .Sum() which also iterates them all. And then another .Sum() for countAll iterates all elements.

How could I optimize this further? I'm sure there is something obvious that I'm missing.

Answer 1

var result = initialGroups
                    .Where(grp => CalculateArg(grp.a.Arg) == item.Key);

This isn't cached.

foreach (var x in result) {} 
foreach (var x in result) {} 
foreach (var x in result) {} 
foreach (var x in result) {} 

will recalculate everything 4 times.

Do it this way:

var result = initialGroups
                    .Where(grp => CalculateArg(grp.a.Arg) == item.Key)
                    .ToArray();

Answer 2

I guess this may improve it partly:

foreach (IGrouping<DateTime, DateTime> item in groups)
{
    var common  =   initialGroups
                    .GroupBy(grp => {
                            var c = CalculateArg(grp.a.Arg);
                            return (c == item.Key && grp.b.Arg == someId) ? 1 :
                                    c == item.Key ? 2 : 3;
                            })
                    .OrderBy(g=>g.Key)
                    .Select(g=>g.Sum(c=>c.Value)).ToList();
    var countMatchId = common[0];
    var countAll = common[0] + common[1];
}

Answer 3

Now there are couple of things we need to consider in this question. First of all, where are your data coming from? Is it coming from an entity that is created by dbcontext? If yes, you need to consider accessing and manipulating your data with Context instead of using a navigation property of objects. What do i mean by that? Consider two classes below,

public class User{

   public int ID { get;set; } 
   public virtual ICollection<Animal> Animals {get;set;} 

}


public class Animal{
    public int ID { get; set; }
    public string Name {get;set;}
    [ForeignKey("Owner")]
    public int? Owner_ID {get;set;}
    public virtual User Owner {get;set;}
}

Now instead of accessing animals of the user with code below,

User user = Context.User.Single(t=> t.ID == 1);
List<Animal> animals = user.Animals.ToList();

accessing with directly dbcontext is much much more efficient. (There is performance considerations should be taking into account if your list have like 100k entity and trying to get it into memory with ToList method.

List<Animal> animals = Context.Animals.Where(t => t.Owner_ID == 1).ToList();

Besides, if you are not using any ORM framework, try getting all the computational objects into memory and cache them all. This will make a huge performance improvement because accessing an object that is already in memory is much easier than the object in a Queryable list. In your case groups object might be a queryable object that's why your performance is not that much good.

Answer 4

If there are lots of item s in groups you may benefit from changing the algorithm around.

Instead of iterating, try calculating things once & GroupJoin the results together, ala

var calculated = initialGroups
  .Select(group => new { Group = group, Arg = CalculateArg(group.a.Arg) })
  .ToList();

var sumCollection = groups
  .GroupJoin(calculated,
             item => item.Key,
             group => group.Arg,
      (group, calculatedCollection) =>
         new {
            Group = group,
            SumAll = calculatedCollection.Sum(y => y.Group.c.Value),
            SumMatchId = calculatedCollection
                         .Where(y => y.Group.b.Arg == someId)
                         .Sum(y => y.Group.c.Value)
         });

foreach (var item in sumCollection)
{
    item.SumAll     // you get the idea
    item.SumMatchId // 
}

Answer 5

I found a way to fix it: following the helpful comments to the question, I profiled almost every line of the foreach with a Stopwatch and found out that indeed, the CalculateArg() function was the culprit - calling it was adding 500ms for each iteration; on a 40 item collection, this meant a total of 20000 ms = 20 seconds.

What I did was to move the calculation outside the foreach, meaning the groups (anonymous object made with SelectMany), now also includes the result of CalculateArg() for each element. which brings the code to:

foreach (IGrouping<DateTime, DateTime> item in groups)
{
    var result = initialGroups
                    .Where(grp => grp.calculatedArg == item.Key);
}