如何在Java中获取170万条记录？

Question

I am using MySQL database in which a table has 1.7 million records. Through Restlet framework in Java I want to fetch these records and return it to the client. I am using Linux Centos which is remote server. I have created WAR file and uploaded on the server. When I run the service it is taking lot of time to respond. I waited for 40 mins but not getting any output. So Can anybody please help me to resolve this problem?

Answer 1

That's probably not going to work: holding that many rows of data in memory will probably cause an out of memory exception (can you look at the logs on the server and see what exactly is happening?).

To do something like this you'll either need to abandon that plan and do pagination of some sort, or you'll need a solution that lets you stream the records to the client without holding them in memory. I'm unsure that the Restlet framework lets you do that: you'll probably need to implement that using servlets yourself.

Answer 2

When I have a very large number of rows I have used memory mapped files. eg I have one database where I have to retrieve and process 1.1 billion rows in around a minute. (Its over 200 GB)

This is a very specialist approach, and I suspect there is a way to tune your SQL database or use a NoSQL database to do what you want. I would have though you can retrieve 1.7 million in under a minute depending on what you are doing (eg if you are selecting this many amoungst a few TB its going to take a while)

But, if there really is no other option, you could write a custom data store.

BTW: Only a summary is produced. No one should be expected to read that many rows, certainly not display them in a browser. Perhaps there is something you can do to produce a report or summary so there is less to send the client.

Answer 3

I have successfully done just this kind of work in my apps. If your client is ready to accept a big response, there is nothing wrong with the approach. The main point is that you'll need to stream the response, which means you can't build te entire response as a string. Get the outputstream of the HTTP response and write records into it one by one. On the db-end you need to set up a scrollable resultset (easy to do at the JDBC level, as well as at the Hibernate level).