Writing huge rows into a file

Question

I have a huge result set including , let's say, 1 million rows. I normally use the following snippet in order to write them into a file, whereas I am always out of memory. Are there any effective ways to eliminate this problem ?

 PrintWriter writer = new PrintWriter(filename, "UTF-8");
 iteration -> { writer.println(a single string); }

 writer.close();

Answer 1

Since you said ResultSet, I take it this is coming from a database? First, use a streaming result set. MySQL's JDBC driver likes to buffer all rows in memory, and with 1M (or 1B) rows, this can be a problem.

If you want this to scale, do it in a threaded way. Have one thread, the producer, reading rows from the ResultSet, and another, the consumer, writing rows to the file. Something like an ArrayBlockingQueue is great for this. If the consumer can't keep up with the producer, start blocking, and wait from the consumer to catch up. This approach takes less memory and is faster, since your IO is being done in parallel.