Java- Writing huge data to csv

Question

I am just trying to write huge data which is fetching from mysql db to CSV by using supercsv. How simply I can manage the performance issue. Does super csv write with some limits?

Answer 1

Since you included almost no detail in your question about how you are approaching the problem, it's hard to make concrete recommendations. So, here's a general one:

Unless you are writing your file to a really slow medium (some old USB stick or something), the slowest step in your process should be reading the data from the database.

There are two general ways how you can structure your program:

1. The bad way: Reading all the data from the database into your application's memory first and then, in a second step, writing it all in one shot to the csv file.
2. The right way: "Stream" the data from the db into the csv file, ie write the data to the csv file as it comes in to your application (record by record or batch by batch).

The idea is to set up something usually referred to as a "pipeline". Think of it like conveyor belt construction in a factory: You have multiple steps in your process of assembling some widget. What you don't want to do is have station 1 process all widgets and have stations 2 and 3 sit idle meanwhile, and then pass the whole container of widgets to station 2 to begin work, while stations 1 and 3 sit idle and so forth. Instead, station 1 needs to send small batches (1 at a time or 10 at a time or so) of widgets that are done to station 2 immediately so that they can start working on it as soon as possible. The goal is to keep all stations as busy as possible at all times.

In your example, station 1 is mysql retrieving the records, station 2 is your application that forwards (and processes?) them, and station 3 is supercsv. So, simply make sure that supercsv can start working as soon as possible, rather than having to wait for mysql to finish the entire request.

If you do this right, you should be able to generate the csv file as quickly as mysql can throw records at you ^* , and then, if it's still too slow, you need to rethink your database backend.

^* I haven't used supercsv yet, so I don't know how well it performs, but given how trivial its job is and how popular it is, I would find it hard to believe that it would end up performing less well (as measured in processing time needed for one record) than mysql in this task. But this might be something that is worth verifying...