How to handle System.OutOfMemoryException when using protobuf-net?

Question

I'm using protobuf-net in ac# application to load and save my program's 'project files'. At save time, the program creates a ProjectData object and adds many different objects to it - see general principle below.

static ProjectData packProjectData()
{
    ProjectData projectData = new ProjectData();

    projectData.projectName = ProjectHandler.projectName;

    foreach (KeyValuePair<int, Module> kvp in DataHandler.modulesDict)
    {
        projectData.modules.Add(serializeModule(kvp.Value));
    }

    return projectData;
} 

[ProtoContract]
public class ProjectData
{
    [ProtoMember(1)]
    public List<SEModule> modules = new List<SEModule>();

    [ProtoMember(2)]
    public string projectName = "";
}

Once this is created, it's zipped and save to the disk. The problem I am having is that when the number of modules gets very big (40,000+) System.OutOfMemoryException is being reported during the packProjectData function.

I've seen questions like this asked already, but these do not contain a clear answer to address the problem. If anyone can give me either a specific solution, or a general principle to follow that would be greatly appreciated.

Answer 1

What sort of size are we talking about here? Most likely this is due to buffering required for the length prefix - something that v3 will address, but for now - if the file is huge, a pragmatic workaround might be:

[ProtoContract]
public class ProjectData
{
    [ProtoMember(1, DataFormat = DataFormat.Grouped)]
    public List<SEModule> modules = new List<SEModule>();

    [ProtoMember(2)]
    public string projectName = "";
}

This changes the internal encoding format of the SEModule items so that no length-prefix is required. This same approach may also be useful for some elements inside SEModule , but I can't see that to comment.

Note that this changes the data layout, so should be considered a breaking change.