Lambda - Nested forEach operations on multiple List<T> very slow

Question

I've got a large operation that I'm trying to accomplish using LINQ by merging multiple collections into one (callQuery in the snippet below) and then updating another list from the merged collection (dataCopy).

The reason the code has been structured this way is to process these calculations in memory (even though some of this may be better suited as a database operation) since the data is coming in from SQL and NoSQL DB sources.

It works but it's horrendously slow. It takes over 10 seconds to complete with the lists of the following sizes:

• dataCopy = 397 rows (this object is to get updated in the loop)
• callQuery = 792 (this object is a projected merge of SQL/NoSQL datasets)
• holidayStructureList = 11 (object contains info on whether a day is a holiday)
• HourValuePairs = 24 (object will always be 24 values and contains hour:value pairs for a particular day)

Am I going about this completely wrong or is there something in my code that's throwing things off?

dataCopy.ForEach(dc =>
            {
                //Apply Cost and Rate to each Hour:Value pair 
                dc.HourValuePairs.ForEach(dchv => {
                      var matchQuery =
                            (
                                //If there were holidays...
                                holidayStructureList != null
                                ?
                                    //If the day is a holiday
                                    (holidayStructureList.Select(h => h.Date.Date).Contains(dc.Date.Date)) 
                                    ?
                                        //Then ensure that the IsHoliday flag is set
                                        callQuery.Where(
                                            cq => cq.Date.Date == dc.Date.Date && cq.PDay.ToLower() == "holiday"
                                            && dc.Date >= cq.StartDate && dc.Date <= cq.EndDate
                                            && dchv.h >= cq.ItemStartHour && dchv.h <= cq.ItemEndHour
                                            && cq.IsHoliday == true
                                        )
                                        :
                                            //Else, it's a regular day
                                            callQuery.Where(
                                                cq => cq.Date.Date == dc.Date.Date && cq.PDay.ToLower() == dc.Date.DayOfWeek.ToString().ToLower()
                                                && dc.Date >= cq.StartDate && dc.Date <= cq.EndDate
                                                && dchv.h >= cq.ItemStartHour && dchv.h <= cq.ItemEndHour
                                                && cq.IsHoliday == false
                                            )
                                    :
                                    //No holidays found in the collection, all days are non-holidays
                                    callQuery.Where(
                                        cq => cq.Date.Date == dc.Date.Date && cq.PDay.ToLower() == dc.Date.DayOfWeek.ToString().ToLower()
                                        && dc.Date >= cq.StartDate && dc.Date <= cq.EndDate
                                        && dchv.h >= cq.ItemStartHour && dchv.h <= cq.ItemEndHour
                                        && cq.IsHoliday == false
                                    )
                            );

                      dchv.Type = matchQuery.Select(x => x.TypeIndexes).FirstOrDefault();
                      dchv.Cost = decimal.Round(matchQuery.Select(x => x.Rate).FirstOrDefault() * (dchv.value == null ? 0 : Convert.ToDecimal(dchv.value)), 2);
                      dchv.Label = matchQuery.Select(x => x.Label).FirstOrDefault();
                });

                //Apply Type Grouping
                var types = dc.HourValuePairs.GroupBy(x => x.Type).Select(y => new
                {
                    Index = y.Key,
                    Value = y.Sum(z => z.value)
                });

                dc.ValueType1 = types.Where(x => x.TypeId == 1).Select(x => x.Value).FirstOrDefault() ?? 0m;
                dc.ValueType2 = types.Where(x => x.TypeId == 2).Select(x => x.Value).FirstOrDefault() ?? 0m;
                dc.ValueType2 = types.Where(x => x.TypeId == 3).Select(x => x.Value).FirstOrDefault() ?? 0m;
            });

Thanks for your help!

EDIT:

Based on Omada's comment, I tried the following and it brought down the query execution to 1.25 seconds.

dataCopy.ForEach(dc =>
            {
                var callQueryWithHolidays = callQuery.Where(
                                            cq => cq.Date.Date == dc.Date.Date && cq.TOUProfileDay.ToLower() == "holiday"
                                            && dc.Date >= cq.EffectiveDate && dc.Date <= cq.EndDate
                                            && cq.IsHoliday == true
                                        ).ToList();

                var callQueryWithoutHolidays = callQuery.Where(
                                                cq => cq.Date.Date == dc.Date.Date && cq.TOUProfileDay.ToLower() == dc.Date.DayOfWeek.ToString().ToLower()
                                                && dc.Date >= cq.EffectiveDate && dc.Date <= cq.EndDate
                                                && cq.IsHoliday == false
                                            ).ToList();

                //Apply Cost and Rate to each Hour:Value pair 
                dc.HourValuePairs.ForEach(dchv => {
                      var matchQuery =
                            (
                                //If there were holidays...
                                holidayStructureList != null
                                ?
                                    //If the day is a holiday
                                    (holidayStructureList.Select(h => h.Date.Date).Contains(dc.Date.Date)) 
                                    ?
                                        //Then ensure that the IsHoliday flag is set
                                        callQueryWithHolidays.Where(
                                            cq => dchv.h >= cq.ItemStartHour && dchv.h <= cq.ItemEndHour
                                        )
                                        :
                                            //Else, it's a regular day
                                            callQueryWithoutHolidays.Where(
                                                cq => dchv.h >= cq.ItemStartHour && dchv.h <= cq.ItemEndHour
                                            )
                                    :
                                    //No holidays found in the collection, all days are non-holidays
                                    callQueryWithoutHolidays.Where(
                                        cq => dchv.h >= cq.ItemStartHour && dchv.h <= cq.ItemEndHour
                                    )
                            );

                      dchv.Type = matchQuery.Select(x => x.TypeIndexes).FirstOrDefault();
                      dchv.Cost = decimal.Round(matchQuery.Select(x => x.Rate).FirstOrDefault() * (dchv.value == null ? 0 : Convert.ToDecimal(dchv.value)), 2);
                      dchv.Label = matchQuery.Select(x => x.Label).FirstOrDefault();
                });

                //Apply Type Grouping
                var types = dc.HourValuePairs.GroupBy(x => x.Type).Select(y => new
                {
                    Index = y.Key,
                    Value = y.Sum(z => z.value)
                });

                dc.ValueType1 = types.Where(x => x.TypeId == 1).Select(x => x.Value).FirstOrDefault() ?? 0m;
                dc.ValueType2 = types.Where(x => x.TypeId == 2).Select(x => x.Value).FirstOrDefault() ?? 0m;
                dc.ValueType2 = types.Where(x => x.TypeId == 3).Select(x => x.Value).FirstOrDefault() ?? 0m;
            });

Answer 1

As in the comments above:

It looks like in your callQuery.Where calls, you are filtering by whether it's a holiday or not. You can do this beforehand outside the dataCopy.ForEach loop and make two filtered lists. Then you won't have to loop over all 792 in the inner loop.

It seems this worked well for you! :) If you need to get it lower than 1.25 seconds you are probably going to have to do what @Alexei Levenkov suggested and start making dictionaries of your data.