如何在Servlet中改善org.apache.commons.io.FileUtils copyInputStreamToFile

Question

I am working on a servlet which does receive attachments in a HTTP POST multipart message. These files have to be stored to a mounted filesystem. It all does work well, only the performance is not what i would expect based on the IO tests i ran on the filesystem.

On my machine i have 2 filesystems, a local filesystem FSA and a CIFS filesystem FSB hosted on another host. The FSB has to be used when going live, FSA i used for comparing the performance of FSB. When i test throughput and latency of the filesystem with the following commands;

througput dd if=/dev/zero of=$FS/test.img bs=1G count=1 oflag=dsync latency dd if=/dev/zero of=$FS/test.img bs=512 count=1000 oflag=dsync

i do get the following readings;

• FSA througput 844MB/s, latency 590MB/s
• FSB througput 50,4MB/s, latency 116KB/s

To save the file to disk i do use the copInputStreamToFile method from org.apache.commons.io.FileUtils. There is no possibility to used JDK7 and higher and therefore java.nio is no option.

// Register starttime of FileIO
startTime = System.currentTimeMillis();

// Fastest way to write stream to file
FileUtils.copyInputStreamToFile(formFieldStream, newFile);
LOG.info(String.format("[Session: %s] [Operation: maakaan io] [File: %s] [%s ms]",
    sessionID, fileNaam,
    (System.currentTimeMillis() - startTime)));

When saving a 110MB file with this servlet to FSA it takes around 10 seconds, on FSB around 13 seconds. So the filesystem of choice makes a difference, but the difference in writing speed does not correspond the measured througput differences. When using the designated filesystem FSB, writing a file from the Java code is with about 10MB/s still much slower then when writing it directly using dd. Since i am using a large file in the test, i chose to use the measured througput of the filesystems.

Are there any options to improve the writing speed of this servlet or how to determine what is causing the 'sluggishnes'

Cheers, Peter

== Update 1 ==

It seems that there is a buffer of 4K in the implementation of FileUtils.copyInputStreamToFile and org.apache.commons.fileupload. I can also see this when making analyzing the flightrecording (JRockit). I have been able to increase the write buffer. Now working to see if i can workaround the buffer in fileUploader.

Answer 1

The I/O tests you ran on the file system using a virtual source are irrelevant. The limiting factor here is the network, which is orders of magnitude slower than the file system, or the CPU, or Java, or IOUtils .