如何根据某些条件从列表中删除项目

Question

I have a list of objects and each object in the list has the following properties:

• FileName,
• SwVersion,
• ControllerSN,
• FirstDoseTime,
• EvtTime,
• Evt,
• EvtOutput

First I need to find all of the objects that have their FirstDoseTime property set to a specific value (1/1/1, 12:00:00). Let's call these blankFirstDose objects.

Then, I need to search through the list and see if for each blankFirstDose object, there is another object in the list that has the same exact properties as the blankFirstDose object except for the FirstDoseTime property (anything other than 1/1/1, 12:00:00) and the FileName property.

Finally, I want to remove any blankFirstDose objects from the list that have a corresponding object in the list that has a different FileName and a FirstDoseTime that is something other than 1/1/1, 12:00:00.

Here is my first pass at doing so. This attempt works successfully, but takes far too long when I have a large list of these objects. I need something that is more efficient and much faster. Thanks for your help!

EDIT: Profiling

I performed some system profiling for a "typical" case to help inform this discussion. In this case, the AllEvents list contains 40,928 LogFileEvent items and the blankFirstDoseLst contains 12,092 items.

The amount of time it takes to complete all of these operations on my system is approximately 16 seconds. Almost all of those 16 seconds are spent in the two foreach loops.

var blankFirstDoseLst = 
    AllEvents.GroupBy(e => e.FirstDoseTime)
             .Where(g => g.FirstOrDefault().FirstDoseTime == PatientModel.DEFAULT_DATE)
             .SelectMany(g => g)
             .ToList();
List<LogFileEvent> blanksToRemove = new List<LogFileEvent>();
foreach (var evt in AllEvents)
{
    foreach (var blankEvt in blankFirstDoseLst)
    {
        if ((evt.FileName      != blankEvt.FileName) &&
            (evt.SwVersion     == blankEvt.SwVersion) &&
            (evt.ControllerSN  == blankEvt.ControllerSN) &&
            (evt.FirstDoseTime != blankEvt.FirstDoseTime) &&
            (evt.EvtTime       == blankEvt.EvtTime) &&
            (evt.Evt           == blankEvt.Evt) &&
            (evt.EvtOutput     == blankEvt.EvtOutput))
        {
            blanksToRemove.Add(blankEvt);
        }
    }
}

// Remove any duplicates ignoring first dose date and time
AllEvents.RemoveAll(e => blanksToRemove.Contains(e));

Answer 1

You could make use of a HashSet for keeping the blanks that should be removed.

var blanksToRemove = new HashSet<LogFileEvent>();

Then the Contains

AllEvents.RemoveAll(e => blanksToRemove.Contains(e));

would be an O(1) operation, avoiding the linear search that it has to be done, when you use a List. In case of using a list the time complexity of Contains it is of the order of O(n), where n is the number of items in your list.

Update

Another change that you could make it is the following. Since you want

...to search through the list and see if for each blankFirstDose object, there is another object in the list that has the same exact properties as the blankFirstDose object except for the FirstDoseTime property (anything other than 1/1/1, 12:00:00) and the FileName property.

You could eliminate the items of the first foreach statement as below:

foreach (var evt in AllEvents.Where(evt=>evt.FirstDoseTime != PatientModel.DEFAULT_DATE))
{
    foreach (var blankEvt in blankFirstDoseLst)
    {
        if ((evt.FileName      != blankEvt.FileName) &&
            (evt.SwVersion     == blankEvt.SwVersion) &&
            (evt.ControllerSN  == blankEvt.ControllerSN) &&
            (evt.EvtTime       == blankEvt.EvtTime) &&
            (evt.Evt           == blankEvt.Evt) &&
            (evt.EvtOutput     == blankEvt.EvtOutput))
        {
            blanksToRemove.Add(blankEvt);
        }
    }
}

Doing so you will avoid to loop through the items in blankFirstDoseLst for the items AllEvents that have as a FirstDoseTime a different than the value you are looking for.

Answer 2

Thank you for all the responses I received. I ended up using a hashset strategy with a custom IEqualityComparer similar to what @ZdeněkJelínek mentioned. The custom IEqualityComparer implements the checks I was previously doing in the "if" statement.

This has significantly reduced the amount of processing time down from 16s to about 1s.

            // Remove any events with blank first dose time if there is a duplicate that has
            // a valid first dose time
            var blanksToRemove        = new List<LogFileEvent>();
            var blankFirstDoseEvts    = AllEvents.Where(e => e.FirstDoseTime == PatientModel.DEFAULT_DATE)
                                                 .ToList();
            var nonBlankFirstDoseEvts = new HashSet<LogFileEvent>(AllEvents.Where(e => e.FirstDoseTime != PatientModel.DEFAULT_DATE),
                                                                  new BlankFirstDoseComparer());

            foreach (var blankEvt in blankFirstDoseEvts)
            {
                if (nonBlankFirstDoseEvts.Contains(blankEvt))
                {
                    blanksToRemove.Add(blankEvt);
                }
            }

            AllEvents.RemoveAll(e => blanksToRemove.Contains(e));