将大数据从DB导出到CSV

Question

I need to export a Table into a CSV File and serve it to download via my JSF/Icefaces Webapplication.

How can I do that? I have a Table with 20+ Columns and over 10 MIO rows.

At the moment, I use a Java Thread, loading all data into ram. Then I create a new File and iterate the Collection writing row for row into the file. If the Thread is done, the user can download the large file via Servlet.

But I dont want to write so many GB into ram. I cant secure, not to get a memory problem..

Is it possible that hibernate does it for me? Or does somebody has an other idea?

Im connected to a DB2 Datebase. The table I want to export is connected to a hibernate bean but it is also possible to write native sql.

Thank you for response!

Answer 1

Do you need the intermediate stage of a file ? Have you tried loading from the database and writing to your servlet output stream for each row ? That way you're acting simply as a pipe between the client and the db.

Simply set your content-disposition header appropriately and that will signal the client's browser to treat the incoming data as a CSV file itself.

Answer 2

I have also gone through with similar kind of problem , the way I solve the problem is that I initially write a CSV file on disc and fetch 25K batch records from DB ans save to the file, and iteratively repeat the process until all the data required by the report is not written on the file. And then send the file URL to the client to download the file.