Downloading the Result set as flat file using Java

Question

Friends,

In my application, i came across an scenario, where the user may request for an Report download as a flat file, which may have max of 17 Lakhs records (around 650 MB) of Data. During this request either my application server stops serving other threads or occurs out of memory exception.

As of now i am iterating through the result set and printing it to the file.

When i Google out for this, i came across an API named OpenCSV. I tried that too but i didn't see any improvement in the performance.

Please help me out on this.

Thanks for the quick response guys, Here i added my code snap

   try {

    response.setContentType("application/csv");

    PrintWriter dout = response.getWriter();

    while(rs.next()) {
     dout.print(data row);  // Here i am printing my ResultSet tubles into flat file.
     dout.print("\r\n");
     dout.flush();
}

Answer 1

OpenCSV will cleanly deal with the eccentricities of the CSV format, but a large report is still a large report. Take a look at the specific memory error, it sounds like you need to increase the Heap or Max Perm Gen space (it will depend of the error to be sure). Without any adjusting the JVM will only occupy s fixed amount of RAM (my experience is this number is 64 MB).

Answer 2

If you only stream the data from resultset to file without using big buffers this should be possible, but maybe you are first collecting the data in a growing list before sending to file? So you should investigate this issue.

Please specify your question more otherwise we have to speculate.

Answer 3

CSV format aren't limited by memory anymore --well, maybe only during prepopulating the data for CSV, but this can be done efficiently as well, for example querying subsets of rows from DB using for example LIMIT/OFFSET and immediately write it to file instead of hauling the entire DB table contents into Java's memory before writing any line. The Excel limitation of the amount rows in one "sheet" will increase to about one million.

Most decent DB's have an export-to-CSV function which can do this task undoubtely much more efficient. In case of for example MySQL, you can use the LOAD DATA INFILE command for this.