使用 newtonsoft json C# 序列化大对象列表

Question

I have my object as below

public class CustomizedObject
{
    public int KeyId { get; set; }
    public string SomeName { get; set; }
    public DateTime StartDate { get; set; }
    public bool isPossible { get; set; }

    //Below dictionary is hug contains more than 300k rows (300*1000) 
    public Dictionary<string, SomBigObject> BigObjectMap { get; set; }

}

public class SomBigObject
{
    //This object has so many properties of all type (int, float, datetime, enum, double, bool) 
    //+ another customized user object Lists
}

I am trying to serialize this object using newtonsoft json library in C# using below settings.

var settings = new JsonSerializerSettings
            {
                PreserveReferencesHandling = PreserveReferencesHandling.Objects,
                NullValueHandling = NullValueHandling.Ignore,
                ReferenceLoopHandling = ReferenceLoopHandling.Serialize,
                Formatting = Formatting.Indented
            };
var json = JsonConvert.SerializeObject(value, settings);

But it gives me Out of memory exception. If i reduce number of rows to small number like 100k rows then it works fine.

When it has given out of memory exception, my computer has still 5 GB RAM available.

After serializing this object, i want to store it in Distributed Cache (Apache Ignite).

Please let me know what alternatives i can try to resolve this issue.

Answer 1

Basically it's not quite clear what are you trying to achieve with the Ignite, but here are my observations. You don't need to serialize your data manually. Ignite is quite smart to perform those things for you. Moreover, it's counterproductive to save the data as a single batch of strings.

You could try to use the data colocation and have two caches - one for the CustomizedObject and the second with your SomBigObject [1] and fill the latter using a DataStreamer [2] . Regarding the serialization, Ignite stores all data using its own binary format [3] in order to provide required APIs, such as SQL for example. For a large custom object, it's a common practice to implement the IBInarizable interface [4] and get benefits out of the GetRawReader and GetRawWriter methods.

Answer 2

Your dictionary can be viewed as an IEnumerable<KeyValuePair<string, SomBigObject>> . Why don't you break that up into chunks of 100K rows and store them one chunk at a time?