使用Java客户端的Java OutputStream文件上传性能

Question

I have a java jar application running in the background of my linux servers (debian 7). The servers receives files through apache generally, then the jar pull the database regularly to upload the files to their final destination using a pool of http connections from an httpclient (v2.5).

The servers are 16gb of ram, 8 cores cpu, 1To disk and 2Gb/s internet bandwidth. My problem is that the bandwidth is only used at 10% or 20% for the uploads made by the jar. After many investigations I think it is because of the capacity of the distant server which might be the bottleneck. So I wanted to launch more threads on my servers to proceed more files at the same time and use all the bandwidth I have, unfortunately file upload with httpclient is eating a lot of cpu it seems !

Actually the jars are working with 20 simultaneous upload runnable threads and the cpu is constantly at 100%, if I try to start more threads the load average increase and break records, getting the system so slow and unusable. Strange thing is the iowaits seems to be null, so I really don't know what is causing the load average.

I have run an hprof using only one thread, here is the result :

CPU SAMPLES BEGIN (total = 4617) Thu Jul 28 17:42:35 2016 rank self accum count trace method 1 52.76% 52.76% 2436 301157 java.net.SocketOutputStream.socketWrite0 2 33.53% 86.29% 1548 300806 java.net.SocketInputStream.socketRead0 3 1.62% 87.91% 75 301138 org.sqlite.core.NativeDB.step 4 1.47% 89.39% 68 301158 java.io.FileInputStream.readBytes 5 1.06% 90.45% 49 300078 java.lang.ClassLoader.defineClass1 6 0.26% 90.71% 12 300781 com.mysql.jdbc.SingleByteCharsetConverter.<clinit> 7 0.19% 90.90% 9 300386 java.lang.Throwable.fillInStackTrace 8 0.19% 91.10% 9 300653 java.lang.ClassLoader.loadClass 9 0.19% 91.29% 9 300780 com.mysql.jdbc.SingleByteCharsetConverter.<clinit> 10 0.17% 91.47% 8 300387 java.net.URLClassLoader.findClass 11 0.17% 91.64% 8 300389 java.util.zip.Inflater.inflateBytes 12 0.15% 91.79% 7 300090 java.lang.ClassLoader.findBootstrapClass 13 0.15% 91.94% 7 300390 java.util.zip.ZipFile.getEntry 14 0.13% 92.07% 6 300805 java.net.PlainSocketImpl.socketConnect

the files are sent with a common httpclient POST execute request with an overrided writeTo() method from the filebody class that uses a bufferedInputStream of 8kb.

Do you think it is possible to reduce the performance impact of the file uploads, and solve my problem of unused bandwidth ?

Thanks in advance for your help.

Answer 1

You should try https://hc.apache.org/

You could change your httpclient library to an library based on java.nio: https://docs.oracle.com/javase/7/docs/api/java/nio/package-summary.html

HttpCore is a set of low level HTTP transport components that can be used to build custom client and server side HTTP services with a minimal footprint. HttpCore supports two I/O models: blocking I/O model based on the classic Java I/O and non-blocking, event driven I/O model based on Java NIO .

Please take a look at https://hc.apache.org/httpcomponents-asyncclient-dev/index.html

Hope this is the right idea.

Answer 2

In fact it appears that the problem was not coming from the java process but from apache consuming all the IO of the hard drive, screwing up the performances. Thanks for your help anyway.