Jetty takes 100% CPU when uploading multipart file and connection breaks

Question

Jetty takes 100% CPU when uploading multipart file and connection breaks

We are currently uploading a file using an embedded Jetty Server + Spring MVC. When I upload manually a file using curl and break it on purpose with Ctrl^C, CPU gets to 100% on server and only rebooting the application will get this problem done.

curl -i -X POST -H "Content-Type: multipart/form-data" -F "param1=bla" -F "file=@a_file" http://myserver/bla/submit

I have checked other posts where this issue is addresed, but generally it gets solved by updating Jetty API version and Java 8. I have done that, but the same issue is happening. Could be related to spring?

APIs:

• Jetty: 9.3.6.v20151106
• JRE: 1.8_65 (I'm not using JDK to run the application)
• Spring MVC: 3.2.15

Here's a code snippet of the method:

@RequestMapping(
            value = MY_PATH,
            method = RequestMethod.POST)
    public
    @ResponseBody
    void submitFile(
            @NotNull @RequestParam("param1") String param1,
            @NotNull @RequestParam("file") MultipartFile file) {
            //No code gets executed
            }

I have used Java Visual VM to monitor and find the Thread dump for the one which is causing the issue:

"qtp529864074-68" - Thread t@68
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
    at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
    at sun.nio.ch.IOUtil.write(IOUtil.java:148)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
    - locked <6ddac168> (a java.lang.Object)
    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:177)
    at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:419)
    at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:313)
    at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:141)
    at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:735)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
    at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:509)
    at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:668)
    at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:722)
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:177)
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:163)
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:441)
    at org.eclipse.jetty.util.ByteArrayISO8859Writer.writeTo(ByteArrayISO8859Writer.java:109)
    at org.eclipse.jetty.server.Response.sendError(Response.java:606)
    at org.eclipse.jetty.server.Response.sendError(Response.java:512)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:650)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
    at org.eclipse.jetty.server.Server.handle(Server.java:517)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
    at java.lang.Thread.run(Thread.java:745)

Answer 1

Looking at the threaddump you'll see a .flush() call from a Response.sendError() call. This could mean that your POST has failed (for whatever reason) and an error response is attempting to be sent, but either hasn't finished yet, or the client hasn't finished reading the error yet. It could even be a result of a sendError loop (where an error page is looked up, used/dispatched to, and that error page itself has an error which causes another sendError to trigger)

As for the Multipart aspect of this, there seems to be a problem with your SpringMVC setup, as there's no problem within Jetty for handling Multipart uploaded content.

An example project for this answer was created on Github for this question.

See https://github.com/jetty-project/multipartconfig-example

An example of doing this in Jetty with the Servlet API.

@MultipartConfig(location="/tmp/upload", 
                 fileSizeThreshold=1024*1024, 
                 maxFileSize=1024*1024*50)
@WebServlet(urlPatterns={"/upload"}, name="upload")
public class UploadServlet extends HttpServlet
{
    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException
    {
        resp.setContentType("text/plain");
        PrintWriter out = resp.getWriter();

        int i=0;
        for(Part part: req.getParts())
        {
            out.printf("Got part: name=%s, size=%d%n",part.getName(), part.getSize());
            part.write(String.format("part-%02d.dat",i++));
        }
    }
}

If we run this in a war file deployed in the Jetty distribution then test it with Curl you'll see the following ...

$ curl -i -X POST -H "Content-Type: multipart/form-data" \
  -F "param1=bla" -F "file=@big.tar.gz" \
  http://localhost:8080/upload
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Content-Type: text/plain;charset=iso-8859-1
Content-Length: 65
Server: Jetty(9.3.6.v20151106)

Got part: name=file, size=23597555
Got part: name=param1, size=3

$ ls -la /tmp/upload/
total 23052
drwxrwxr-x.  2 joakim joakim       80 Jan 26 14:19 .
drwxrwxrwt. 20 root   root        500 Jan 26 14:19 ..
-rw-rw-r--.  1 joakim joakim 23597555 Jan 26 14:19 part-00.dat
-rw-rw-r--.  1 joakim joakim        3 Jan 26 14:19 part-01.dat

$ sha1sum /tmp/upload/part-00.dat big.tar.gz 
13be13872d0203bae2f8a83ad457f8a0d259a2db  /tmp/upload/part-00.dat
13be13872d0203bae2f8a83ad457f8a0d259a2db  big.tar.gz

$ file /tmp/upload/part-01.dat 
/tmp/upload/part-01.dat: ASCII text, with no line terminators

$ cat /tmp/upload/part-01.dat 
bla