protobuf-net collection serialization throws StackOverflowException

Question

I have an embedded C# application using protobuf-net serialization. When serializing a collection of some 50 entries it throws a StackOverflowException, reproduceable only on the device which runs WinCE. The stack is some 600 entries deep and ends like this:

at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)
   at ProtoBuf.Meta.ValueMember.TryGetCoreSerializer(RuntimeTypeModel model, DataFormat dataFormat, Type type, WireType& defaultWireType, Boolean asReference, Boolean dynamicType, Boolean overwriteList, Boolean allowComplexTypes)
   at ProtoBuf.Meta.ValueMember.BuildSerializer()
   at ProtoBuf.Meta.ValueMember.get_Serializer()
   at ProtoBuf.Meta.MetaType.BuildSerializer()
   at ProtoBuf.Meta.MetaType.get_Serializer()
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.Serializers.SubItemSerializer.ProtoBuf.Serializers.IProtoSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.PropertyDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.BclHelpers.WriteNetObject(Object value, ProtoWriter dest, Int32 key, NetObjectOptions options)
   at ProtoBuf.Serializers.NetObjectSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.FieldDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.Serializers.SubItemSerializer.ProtoBuf.Serializers.IProtoSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.ListDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.PropertyDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.BclHelpers.WriteNetObject(Object value, ProtoWriter dest, Int32 key, NetObjectOptions options)
   at ProtoBuf.Serializers.NetObjectSerializer.Write(Object value, ProtoWriter dest)

The list is defined like this:

[ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
        //[ProtoMember(2, AsReference = true)]
        public NodeList<T> Nodes { get; private set; }

(I tried both versions, with and without DataFormat.Group)

[ProtoContract(ImplicitFields = ImplicitFields.AllFields)]        
public class NodeList<T> : Collection<GraphNode<T>> where T : new()

The exception is not thrown when running the app on Windows 7 with more resources.

Is there any possibility to optimize the serialization? Am I doing something wrong?

Why is the call stack so long? I supposed a list should be treated sequential as an array, not by recursive calls?

Thanks, Daiana

Answer 1

You don't include an example of your GraphNode<T> in your question. However, from the name it would appear to a general graph node element, similar to GraphNode<T> , NodeList<T> and Graph<T> from the Microsoft technical article An Extensive Examination of Data Structures Using C# 2.0: Part 5: From Trees to Graphs .

In that case, if your node graph is very deep and the root node is at the beginning of the collection, then it is quite possible to exceed the max stack depth serializing the list of nodes. The reason for this is explained in Protobuf-net: the unofficial manual :

How references work

When serializing a class Foo to which AsReference or AsReferenceDefault applies, the type of the field in the protocol buffer changes from Foo to bcl.NetObjectProxy, which is defined as follows in the source code of protobuf-net (Tools/bcl.proto):

 message NetObjectProxy { // for a tracked object, the key of the **first** // time this object was seen optional int32 existingObjectKey = 1; // for a tracked object, a **new** key, the first // time this object is seen optional int32 newObjectKey = 2; // for dynamic typing, the key of the **first** time // this type was seen optional int32 existingTypeKey = 3; // for dynamic typing, a **new** key, the first time // this type is seen optional int32 newTypeKey = 4; // for dynamic typing, the name of the type (only // present along with newTypeKey) optional string typeName = 8; // the new string/value (only present along with // newObjectKey) optional bytes payload = 10; }

So it appears that

• The first time an object is encountered, the newObjectKey and a payload fields are written; presumably, the payload is stored as if its type is Foo.
• When the object is encountered again, just the existingObjectKey is written.

Thus, if the graph happens to be very deep and the root is encountered as the first element in the list, protobuf-net will recursively traverse the graph depth-first rather than iterating through the array breadth-first, and thereby overflow the stack.

To avoid this problem, you can take advantage of the fact that, when AsReference = true , protobuf-net only iterates though the members of an object the first time it is encountered, to serialize your NodeList<T> in two stages:

1. Serialize the list of nodes without their neighbor information .
2. Then serialize a table of neighbor information.

For instance, using simplified versions of GraphNode<T> , NodeList<T> and Graph<T> from the article above as a basis, this could be accomplished as follows:

[ProtoContract]
public class GraphNode<T> where T : new()
{
    readonly NodeList<T> neighbors = new NodeList<T>();

    public GraphNode()
    {
        this.Value = new T();
    }

    public GraphNode(T value)
        : this()
    {
        this.Value = value;
    }

    [ProtoMember(1, AsReference = true)]
    public T Value { get; set; }

    // Do not serialize the list of neighbors directly!  
    // Instead this will be serialized by the NodeList<T> owned by the Graph<T>
    [ProtoIgnore]
    public NodeList<T> Neighbors { get { return neighbors; } }
}

[ProtoContract(IgnoreListHandling = true)]
public class NodeList<T> : Collection<GraphNode<T>> where T : new()
{
    [ProtoContract]
    class GraphNodeNeighborsProxy
    {
        [ProtoMember(1, AsReference = true)]
        public GraphNode<T> Node { get; set; }

        [ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
        public ICollection<GraphNode<T>> Neighbors
        {
            get
            {
                return Node == null ? null : Node.Neighbors;
            }
        }
    }

    [ProtoMember(1, AsReference = true, DataFormat = DataFormat.Group)]
    IEnumerable<GraphNode<T>> Nodes
    {
        get
        {
            return new SerializationCollectionWrapper<GraphNode<T>, GraphNode<T>>(this, n => n, (c, n) => c.Add(n)); 
        }
    }

    [ProtoMember(2, DataFormat = DataFormat.Group)]
    IEnumerable<GraphNodeNeighborsProxy> NeighborsTable
    {
        get
        {
            return new SerializationCollectionWrapper<GraphNode<T>, GraphNodeNeighborsProxy>(
                this,
                n => new GraphNodeNeighborsProxy { Node = n },
                (c, proxy) => {}
                );
        }
    }
}

[ProtoContract]
public class Graph<T> where T : new()
{
    readonly private NodeList<T> nodeSet = new NodeList<T>();

    public Graph() { }

    public GraphNode<T> AddNode(GraphNode<T> node)
    {
        // adds a node to the graph
        nodeSet.Add(node);
        return node;
    }

    public GraphNode<T> AddNode(T value)
    {
        // adds a node to the graph
        return AddNode(new GraphNode<T>(value));
    }

    public void AddDirectedEdge(GraphNode<T> from, GraphNode<T> to)
    {
        from.Neighbors.Add(to);
    }

    [ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
    public NodeList<T> Nodes
    {
        get
        {
            return nodeSet;
        }
    }
}

public class SerializationCollectionWrapper<TFrom, TTo> : ICollection<TTo>
{
    readonly ICollection<TFrom> collection;
    readonly Func<TFrom, TTo> mapTo;
    readonly Action<ICollection<TFrom>, TTo> add;

    public SerializationCollectionWrapper(ICollection<TFrom> collection, Func<TFrom, TTo> mapTo, Action<ICollection<TFrom>, TTo> add)
    {
        if (collection == null || mapTo == null || add == null)
            throw new ArgumentNullException();
        this.collection = collection;
        this.mapTo = mapTo;
        this.add = add;
    }

    ICollection<TFrom> Collection { get { return collection; } }

    #region ICollection<TTo> Members

    public void CopyTo(TTo[] array, int arrayIndex)
    {
        foreach (var item in this)
            array[arrayIndex++] = item;
    }

    public int Count
    {
        get { return Collection.Count; }
    }

    public bool IsReadOnly
    {
        get { return Collection.IsReadOnly; }
    }

    public void Add(TTo item)
    {
        add(Collection, item);
    }

    public void Clear()
    {
        throw new NotImplementedException();
    }

    public bool Contains(TTo item)
    {
        throw new NotImplementedException();
    }

    public bool Remove(TTo item)
    {
        throw new NotImplementedException();
    }

    #endregion

    #region IEnumerable<TTo> Members

    public IEnumerator<TTo> GetEnumerator()
    {
        foreach (var item in Collection)
            yield return mapTo(item);
    }

    #endregion

    #region IEnumerable Members

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    #endregion
}

Note that this only works if the root node list contains all nodes in the graph. If you do not have a table of all nodes in the graph, you will need to compute the transitive closure of the graph to serialize all nodes up front.