What is the fastest way to output a large amount of data?

Question

I have an JAX-RS web service that calls a db2 z/os database and returns about 240mb of data in a resultset. I am then creating an OutputStream to send this data to the client by looping through the resultset and adding a few XML tags for my output.

I am confused about what to use PrintWriter, BufferedWriter or OutputStreamWriter. I am looking for the fastest way to deliver the data. I also don't want the JVM to hold onto this data any longer than it needs to, so I don't use up it's memory.

Any help is appreciated.

Answer 1

You should use

1. BufferedWriter
2. Call .flush() frequently
3. Enable gzip for best compression
4. Start thinking about a different way of doing this. Can your data be paginated? Do you need all the data in one request.

Answer 2

If you are sending a large binary data, you probably don't want to use xml. When xml is used, binary data is usually represented using base64 which becomes larger than the original binary and uses quite a lot of CPU for the conversion into base64.

If I were you, I'd send the binary separate from the xml. If you are using WebService, MTOM attachment could help. Otherwise you could send the reference to the binary data in the xml, and let the app. download the binary data separately.

As for the fastest way to send binary, if you are using weblogic, just writing on the response's outputstram would be ok. That output stream is most probably buffered and whatever you do probably won't change the performance anyways.

Turning on gzip could also help depending on what you are sending (eg if you are sending jpeg (stuff that is already compressed) or something, it won't help a lot but if you are sending raw text then it can help a lot, etc.).

Answer 3

One solution (which might not work for you) is to spawn a job / thread that creates a file and then notifies the user when the file is ready to download, in this way you're not tied to the bandwidth of the client connection (and you can even compress the file properly, before the client downloads it)

Some Business Intelligence and data crunching applications do this, specially if the process takes some time to generate the data.

Answer 4

The output max speed will me limited by network bandwith and i am shure any Java OutputStream will be much more faster than you will notice the difference.

The choice depends on the data to send: is that text (lines) PrintWriter is easy, is that a byte array take OutputStream.

To hold not too much data in the buffers you should call flush() any x kb maybe.

Answer 5

You should never use PrintWriter to output data over a network. First of all, it creates platform-dependent line breaks. Second, it silently catches all I/O exceptions, which makes it hard for you to deal with those exceptions.

And if you're sending 240 MB as XML, then you're definitely doing something wrong. Before you start worrying about which stream class to use, try to reduce the amount of data.

EDIT:

The advice about PrintWriter (and PrintStream) came from a book by Elliotte Rusty Harold . I can't remember which one, but it was a few years ago. I think that ServletResponse.getWriter() was added to the API after that book was written - so it looks like Sun didn't follow Rusty's advice. I still think it was good advice - for the reasons stated above, and because it can tempt implementation authors to violate the API contract in order to get predictable behavior.