从对象列表生成CSV

Question

I am trying to generate a CSV from the results of a hibernate query, it could contain up to 200k rows, and when created the CSV could be around 30MB.

I am struggling to get the performance anything near acceptable. Currently my hibernate query will return a list of objects. I am then using a StringBuilder to generate a CSV string. Then I get a byte array from that, and pass the byte array into response.outputstream.write() method.

What should I be doing in order to generate a CSV of this size efficiently?

My Service

List<Object> objectList = hibernateDao.getObjectList(objectList);

    StringBuilder sb = new StringBuilder();
    sb.append("field1_id,field2,field3,field14");
    for(Object object : objectList){
        sb.append("\n");
        sb.append(object.field1());
        sb.append(",");
        sb.append(object.field2());
        sb.append(",");
        sb.append(object.field3());
        ...
        ...
        sb.append(",");
        sb.append(object.field14);
    }
    byte[] bytes = sb.toString().getBytes();
    return bytes;

My Controller will get the resulting bytes[]

response.setHeader("Content-Type", "text/comma-separated-values");
response.getOutputStream().write(objectService.getListByteArray());

Answer 1

The problem initially appears to be on your StringBuilder expanding, you could initialize it to allocate big number of characters but then the length of the content you are trying to write might go over Integer.MAX_VALUE (the maximum length of a char[] ) and you'll get a buffer overflow error.

There's also the potential problem of escaping commas and quote characters in the values you are writing individually. This is tricky to handle and I suggest you to use a CSV library to make sure you get the correct output.

Try uniVocity-parsers to persist your objects. You can use a few annotations to serialize specific fields to CSV.

Disclaimer: I'm the author of this library, it's open-source and free (Apache 2.0 license)