使用 Newton Soft 从大约 5GB 的文件中反序列化非常大的复杂 JSON 数据

Question

I am getting a huge json data file of around 5 to 8 GB in size with following data which consist of company info and then array of employee details. In 1 file, only 1 company info is coming and that file size ranges from 5GB - 8GB

I am trying to de serialize this to c# object. I cannot add this whole data to a string as it will throw memory exception.

I am using NewtonSoft Json

Sample data

{
"Companyname": "ABC Company",
"email": "info@abc.com",
"location": "NYC",
"department": [
    {
        "deptid": "15345",
        "deptname":"dept1",
        "projects": ["25A","26B","26C"],

        "employees":
          [
            {
             "empid": "1",
             "name":"john",
             "groupnumber":[234234,34243,343242,2342342]
            },
            {
             "empid": "2",
             "name":"Joseph",
             "groupnumber":[13245646,78945651,45641546,78978979]
            }
          ]
    },
    {
        "deptid": "5654",
        "deptname":"dept2",
        "projects": ["125A","226B","26CD"],

        "employees":
          [
            {
             "empid": "11",
             "name":"Jill",
             "groupnumber":[13224231,123133333,8765433,213132333]
            },
            {
             "empid": "122",
             "name":"Don",
             "groupnumber":[12344,123123234]
            }
          ]
    }   
]}

Class

public class CompanyDetails
{
    public string companyName{ get; set; }
    public string email { get; set; }
    public string location { get; set; }
    public List<Department> department { get; set; }
}

public class Department
{
    public string deptname { get; set; }
    public int deptid{ get; set; }
    public List<Project> projects{ get; set; }
    public List<Employee> employees{ get; set; }
}

public class Project
{
    public string projectReference { get; set; }
}

public class Department
{
    public int empid { get; set; }
    public string name { get; set; }
    public List<GroupNumber> groupnumber { get; set; }
}

public class GroupNumber
{
public long grpnumber { get; set; }
}

Below is my c sharp code. It's not throwing any error. But the companyData object is empty

 using (StreamReader reader = new StreamReader(file.FullName))
           {
              var serializer = new JsonSerializer();
              CompanyDetails companyData = (CompanyDetails)serializer.Deserialize(reader, typeof(CompanyDetails));
           }

Any help is much appreciated.

Answer 1

You have to fix the classes. For example

 public List<Project> projects{ get; set; }

should be

public List<string> projects { get; set; }

I've been using these classes and everything is working properly

public class CompanyDetails
{
    public string Companyname { get; set; }
    public string email { get; set; }
    public string location { get; set; }
    public List<Department> department { get; set; }
}

public class Department
{
    public string deptid { get; set; }
    public string deptname { get; set; }
    public List<string> projects { get; set; }
    public List<Employee> employees { get; set; }
}

public class Employee
{
    public string empid { get; set; }
    public string name { get; set; }
    public List<int> groupnumber { get; set; }
}

Answer 2

1. Your property names in your C# models need to match keys in the JSON exactly for it to deserialize correctly.

2. You need to change your code a bit inside the using statement to correctly deserialize the file text into your C# model. See this link https://www.newtonsoft.com/json/help/html/Performance.htm#MemoryUsage shared by @dbc in the comments below.

3. Other than that, you're doing great.

EDIT Explained

So as to avoid being raked over the coals again, I overhauled this answer. Apparently, there are some passionate feelings about efficiency of memory usage and directly applicable solutions. Forgive the non-answer and misleading direction offered before.