What is the default concrete type of IEnumerable

Question

(Sorry for the vague title; couldn't think of anything better. Feel free to rephrase.)

So let's say my function or property returns an IEnumerable<T> :

public IEnumerable<Person> Adults
{
  get
  {
    return _Members.Where(i => i.Age >= 18);
  }
}

If I run a foreach on this property without actually materializing the returned enumerable:

foreach(var Adult in Adults)
{
  //...
}

Is there a rule that governs whether IEnumerable<Person> will be materialized to array or list or something else?

Also is it safe to cast Adults to List<Person> or Array without calling ToList() or ToArray() ?

Edit

Many people have spent a lot of effort into answering this question. Thanks to all of them. However, the gist of this question still remains unanswered. Let me put in some more details:

I understand that foreach doesn't require the target object to be an array or list. It doesn't even need to be a collection of any kind. All it needs the target object to do is to implement enumeration. However if I place inspect the value of target object, it reveals that the actual underlying object is List<T> (just like it shows object (string) when you inspect a boxed string object). This is where the confusion starts. Who performed this materialization? I inspected the underlying layers ( Where() function's source) and it doesn't look like those functions are doing this.

So my problem lies at two levels.

• First one is purely theoretical. Unlike many other disciplines like physics and biology, in computer sciences we always know precisely how something works (answering @zzxyz's last comment); so I was trying to dig about the agent who created List<T> and how it decided it should choose a List and not an Array and if there is a way of influencing that decision from our code.
• My second reason was practical. Can I rely on the type of actual underlying object and cast it to List<T> ? I need to use some List<T> functionality and I was wondering if for example ((List<Person>)Adults).BinarySearch() is as safe as Adults.ToList().BinarySearch() ?

I also understand that it isn't going to create any performance penalty even if I do call ToList() explicitly. I was just trying to understand how it is working. Anyway, thanks again for the time; I guess I have spent just too much time on it.

Answer 1

In general terms all you need for a foreach to work is to have an object with an accessible GetEnumerator() method that returns an object that has the following methods:

void Reset()
bool MoveNext()
T Current { get; private set; } // where `T` is some type.

You don't even need an IEnumerable or IEnumerable<T> .

This code works as the compiler figures out everything it needs:

void Main()
{
    foreach (var adult in new Adults())
    {
        Console.WriteLine(adult.ToString());
    }
}

public class Adult
{
    public override string ToString() => "Adult!";
}

public class Adults
{
    public class Enumerator
    {
        public Adult Current { get; private set; }
        public bool MoveNext()
        {
            if (this.Current == null)
            {
                this.Current = new Adult();
                return true;
            }
            this.Current = null;
            return false;
        }
        public void Reset() { this.Current = null; }
    }
    public Enumerator GetEnumerator() { return new Enumerator(); }
}

Having a proper enumerable makes the process work more easily and more robustly. The more idiomatic version of the above code is:

public class Adults
{
    private class Enumerator : IEnumerator<Adult>
    {
        public Adult Current { get; private set; }

        object IEnumerator.Current => this.Current;

        public void Dispose() { }

        public bool MoveNext()
        {
            if (this.Current == null)
            {
                this.Current = new Adult();
                return true;
            }
            this.Current = null;
            return false;
        }

        public void Reset()
        {
            this.Current = null;
        }
    }
    public IEnumerator<Adult> GetEnumerator()
    {
        return new Enumerator();
    }
}

This enables the Enumerator to be a private class, ie private class Enumerator . The interface then does all of the hard work - it's not even possible to get a reference to the Enumerator class outside of Adults .

The point is that you do not know at compile-time what the concrete type of the class is - and if you did you may not even be able to cast to it.

The interface is all you need, and even that isn't strictly true if you consider my first example.

If you want a List<Adult> or an Adult[] you must call .ToList() or .ToArray() respectively.

Answer 2

There is no such thing as a default concrete type for any interface.
The entire point of an interface is to guarantee properties, methods, events or indexers, without the user need of any knowledge of the concrete type that implements it.

When using an interface, all you can know is the properties, methods, events and indexers this interface declares, and that's all you actually need to know. That's just another aspect of encapsulation - same as when you are using a method of a class you don't need to know the internal implementation of that method.

To answer your question in the comments:

who decides that concrete type in case we don't, just as I did above?

That's the code that created the instance that's implementing the interface. Since you can't do var Adults = new IEnumerable<Person> - it has to be a concrete type of some sort.

As far as I see in the source code for linq's Enumerable extensions - the where returns either an instance of Iterator<TSource> or an instance of WhereEnumerableIterator<TSource> . I didn't bother checking further what exactly are those types, but I can pretty much guarantee they both implement IEnumerable , or the guys at Microsoft are using a different c# compiler then the rest of us... :-)

Answer 3

The following code hopefully highlights why neither you nor the compiler can assume an underlying collection:

public class OneThroughTen : IEnumerable<int>
{
    private static int bar = 0;
    public IEnumerator<int> GetEnumerator()
    {
        while (true)
        {
            yield return ++bar;
            if (bar == 10)
                { yield break; }
        }
    }
    IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator(); }
}

class Program
{
    static void Main(string[] args)
    {
        IEnumerable<int> x = new OneThroughTen();
        foreach (int i in x)
            { Console.Write("{0} ", i); }
    }
}

Output being, of course:

1 2 3 4 5 6 7 8 9 10

Note, the code above behaves extremely poorly in the debugger. I don't know why. This code behaves just fine:

public IEnumerator<int> GetEnumerator()
{
  while (bar < 10)
  {
    yield return ++bar;
  }
  bar = 0;
}

(I used static for bar to highlight that not only does the OneThroughTen not have a specific collection, it doesn't have any collection, and in fact has no instance data whatsoever. We could just as easily return 10 random numbers, which would've been a better example, now that I think on it :))

Answer 4

From your edited question and comments it sounds like you understand the general concept of using IEnumerable, and that you cannot assume that "a list object backs all IEnumerable objects". Your real question is about something that has confused you in the debugger, but we've not really been able to understand exactly what it is you are seeing. Perhaps a screenshot would help?

Here I have 5 IEnumerable<int> variables which I assign in various ways, along with how the "Watch" window describes them. Does this show the confusion you are having? If not, can you construct a similarly short program and screenshot that does?

Answer 5

Coming a bit late into the party here:) Actually Linq's "Where" decides what's going to be the underlying implementation of IEnumerable's GetEnumerator.

Look at the source code: https://github.com/do.net/runtime/blob/918e6a9a278bc66fb191c43d4db4a71e63ffad31/src/libraries/System.Linq/src/System/Linq/Where.cs#L59

You'll see that based on the "source" type, the methods return " WhereSelect Array Iterator " or " WhereSelect List Iterator " or a more generic " WhereSelectEnumerableSelector ".

Each of this objects implement the GetEnumerator over an Array, or a List, so I'm pretty sure that's why you see the underlying object type being one of these on VS inspector.

Hope this helps clarifying.

Answer 6

I have been digging into this myself. I believe the 'underlying type' is an iterator method, not an actual data structure type.

An iterator method defines how to generate the objects in a sequence when requested.

https://learn.microsoft.com/en-us/do.net/csharp/iterators#enumeration-sources-with-iterator-methods

In my usecase/testing, the iterator is System.Linq.Enumerable.SelectManySingleSelectorIterator. I don't think this is a collection data type. It is a method that can enumerate IEnumerables.

Here is a snippet:

public IEnumerable<Item> ItemsToBuy { get; set; }

...

    ItemsToBuy = Enumerable.Range(1, rng.Next(1, 20))
        .Select(RandomItem(rng, market))
        .SelectMany(e => e);

The property is IEnumerable and.SelectMany returns IEnumerable. So what is the actual collection data structure? I don't think there is one in how I am interpreting 'collection data structure'.

Also is it safe to cast Adults to List or Array without calling ToList() or ToArray()?

Not for me. When attempting to cast ItemsToBuy collection in a foreach loop I get the following runtime exception:

{"Unable to cast object of type 'SelectManySingleSelectorIterator 2[System.Collections.Generic.IEnumerable 1[CashMart.Models.Item],CashMart.Models.Item]' to type 'CashMart.Models.Item[]'."}

So I could not cast, but I could.ToArray(). I do suspect there is a performance hit as I would think that the IEnumerable would have to 'do things' to make it an array, including memory allocation for the array even if the entities are already in memory.

However if I place inspect the value of target object, it reveals that the actual underlying object is List

This was not my experience and I think it may depend on the IEnumerable source as well as the LinQ provider. If I add a where, the returned iterator is:

System.Linq.Enumerable.WhereEnumerableIterator

I am unsure what your _Member source is, but using LinQ-to-Objects, I get an iterator. LinQ-to-Entities must call the database and store the result set in memory somehow and then enumerate on that result. I would doubt that it internally makes it a List, but I don't know much. I suspect instead that _Members may be a List somewhere else in your code thus, even after the.Where, it shows as a List.