Export large amount of data in JAVA EE Application

Question

I am working on search based Java EE application, It is very simple just like any search engine there, one page which allows the user to search the content and pagination to move to next set OF result.now I want provide the functionality exporting the data in text file but the problem is when there is very large data to be exported it consumes about 100MB of JVM memory.

Java EE experts please advice ?

I had look here but that is not much helpful to me

Answer 1

You are most likely saving the whole data in some kind of collection and then serializing it to a text file. This - as you have noticed - might work for small amounts of data but get your application on its knees when the amount of data is too large. In this case not a good way of doing it because you never how much data a query will return.

You should stream the results of your query to the output, ie write each query result write aftre fetching it, no need for temporarly saving it. You might use any Implementation of OutputStream. If you are sending the file from a servlet you might want to write directly to the outputstream of your servlet after setting the right content type.

Answer 2

100 MB per session are affordable with few simultaneous users. Therefore, you could throttle the number of concurrent export operations to ensure you don't run out of memory;

There are many ways to do it, including a limited pool of Session EJB instances (other request wait automatically), a counter in a static variable of the relevant class, a database table (which can double as a log of past export operations), possibly monitoring the available memory.