查找列表中唯一成员数量的最快方法

Question

I've been trying to find a good way of looking for the number of unique values from a list. There was a very good question here which I tried to peruse to create a solution that looks like this:

gridStats[0] = gridList.SelectMany(x => x.Position.Easting).Distinct().ToList().Count();
gridStats[1] = gridList.SelectMany(x => x.Position.Northing).Distinct().ToList().Count();

However, that seems to produce an error saying that I am implicitly declaring the type arguments that didn't make sense. Further research seemed to suggest that 'Distinct', good as it is, would not actually provide what I am looking for in any case without some additional code.

Therefore, I gave up on that approach and tried to go for a loop method, and I have arrived at this:

List<double> eastings = new List<double>();
List<double> northings = new List<double>();

for (int i = 0; i < gridList.Count; i++)
{
    if (!eastings.Contains(gridList[i].Position.Easting))
    {
        eastings.Add(gridList[i].Position.Easting);
    }

    if (!northings.Contains(gridList[i].Position.Northing))
    {
        northings.Add(gridList[i].Position.Northing);
    }
}

gridStats[0] = eastings.Count;
gridStats[1] = northings.Count;

Note here that 'gridList' can have hundreds of millions of entries.

Quite predictably, this loop is not particularly fast in use. Therefore, I was hoping it would be possible to either get assistance in making that loop more efficient or assistance in sorting out the Linq approach.

What do you suggest as the best approach when the only concern is the speed at which this task is performed?

Answer 1

You were so close.

Distinct is indeed the best choice for this scenario - it's similar to HashSet<T> based implementation, but uses internally a special lightweight hash set implementation. In practice I don't think there will be a noticeable difference in performance, but still Distinct is more readable and at the same time a bit faster.

What you've missed though is that the question in the link is about list of objects having a list property so it needed SelectMany , while in your case the objects hold a single property , so a simple Select will do the job, like this

gridStats[0] = gridList.Select(x => x.Position.Easting).Distinct().Count();
gridStats[1] = gridList.Select(x => x.Position.Northing).Distinct().Count();

Also note that ToList call was not needed in order to use Count extension method. Every operation has a cost, so don't include unnecessary methods - they'll not make your code more readable, but for sure will make it slower and more space consuming.

Answer 2

You can speed this up by using HashSet instead of List for eastings and northings :

HashSet<double> eastings = new HashSet<double>();
HashSet<double> northings = new HashSet<double>();

The reason this would be faster is because a HashSet uses a hash to give O(1) look ups, versus using List which will be O(n) (it has to search the whole list to see if the item exists).