如何使用Java将大型Json数据下载到CSV文件中？

Question

I am new to Java web application. I tried below link for converting JSON array of data to CSV file http://jsfiddle.net/JXrwM/11322/

Now the issue is, whenever getting large amount of JSONArray of value, unable to convert CSV file as soon as possible. Loading time is too long.

Fetching JSONArray like below,

ResultSet rs = (ResultSet) statement.getObject(P_CURSOR);
JSONArray jsonArr = new JSONArray();
while (rs.next()) {
       JSONObject data = new JSONObject();           
       data.put(LASTNAME, rs.getString(LASTNAME));
       data.put(FIRSTNAME, rs.getString(FIRSTNAME));
       jsonArr.put(data); //**Fetching JSONArray is too slow.**
}

Here the data contains multiple records, how to fetch jsonArr faster and download into CSV file from ResultSet data.

Answer 1

Your problem could be that you are doing a "full" read, and then a "full" write to the CSV file.

Meaning: your code first reads all the data into a single ResultSet instance, and that big object is then send to a CSVWriter in one shot.

Maybe the answer is to rework your whole code to do that for rows , one by one.

Meaning:

• you setup your CSVWriter
• you fetch one row from the database
• you write one line using the CSVWriter

Coming from such a solution, you might be able to use different threads for producing and consuming that data. As in: one thread reads single lines, and puts that data on some "queue", and another thread reads from that queue, to write CSV data.

Most likely, that is the only way that would enable you to speed up overall execution time. Of course, using multiple threads will only pay out if there is really enough data to process.

Answer 2

Found solution for write large data in json format from resultset using Jackson library like below,

        ResultSet rs = (ResultSet) statement.getObject(P_CURSOR);

        StringWriter writer = new StringWriter();
        JsonFactory jsonfactory = new JsonFactory();
        JsonGenerator jsonGenerator = jsonfactory.createJsonGenerator(writer);

        jsonGenerator.writeStartArray();
        while (rs.next()) {
            jsonGenerator.writeStartObject();
            jsonGenerator.writeStringField(KEY1, rs.getString(KEY1));
            jsonGenerator.writeStringField(KEY2, rs.getString(KEY2));
            jsonGenerator.writeStringField(KEY3, rs.getString(KEY3));
            jsonGenerator.writeEndObject();
        }

        jsonGenerator.writeEndArray();
        jsonGenerator.close();

This helps to read and write large set of data from resultset with less loading time.