Spring + Jetty：优雅的关机

Question

Objective: Having Spring web application with an embedded Jetty, I'd like to shutdown/restart the application gracefully.
Here's the EmbeddedServletContainerFactory bean (I'll add the omitted code if needed):

@Bean
public EmbeddedServletContainerFactory jettyEmbeddedServletContainerFactory() {
    JettyEmbeddedServletContainerFactory factory = new JettyEmbeddedServletContainerFactory();
    factory.addServerCustomizers(server -> {

        server.setStopAtShutdown(false);

        /*
         * StatisticsHandler has to be added for graceful shutdown to work (see
         * https://github.com/eclipse/jetty.project/issues/1549#issuecomment-301102535)
         */
        StatisticsHandler statisticsHandler = new StatisticsHandler();
        statisticsHandler.setHandler(server.getHandler());
        server.setHandler(statisticsHandler);
    });
    return factory;
}

and here's our shutdown signal handler:

@Component
public class ShutdownSignalHandler {

    @Value("${shutdown.signalType:TERM}")
    private String signal;

    @Autowired
    private ConfigurableApplicationContext context;

    @Autowired
    private Server jetty; 

    @PostConstruct
    public void init() {
        Signal.handle(new Signal(signal), this::signalHandler);
    }

    private void signalHandler(Signal signal) {
        jetty.stop();
        context.close();
    }
}

Question: having the above configuration, every time I restart the application via TERM signal, I see a lot of 503 entries in apache_access log (almost all of them happen while jetty.stop is being executed).
Any idea how/why it happens and what the resolution is?

Answer 1

Why/how does it happen? Well it's the default behavior of Jetty: First, Jetty close the network connectors to stop accepting new connections and then wait for active request to be process. But the catch is new requests might be sent to Jetty over already-established connections (those which are connected before shutdown process starts).
Solutions: Send retry response for new requests sent over open connections. We customized StatisticsHandler so that, during stop process instead of handling request with 503 , it handles them with 307 Connection: close, Location: <original-url> .

UPDATE: what we did was to change the implementation of the handle method of the StatisticsHandler class (Look for Change this line! comment):

@Override
public void handle(String path, Request baseRequest, HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException
{
    ...
    try
    {
        Handler handler = getHandler();
        if (handler!=null && _shutdown.get()==null && isStarted())
            handler.handle(path, baseRequest, request, response);
        else if (baseRequest.isHandled())
        {
            if (_wrapWarning.compareAndSet(false,true))
                LOG.warn("Bad statistics configuration. Latencies will be incorrect in {}",this);
        }
        else
        {
            baseRequest.setHandled(true);
            response.sendError(HttpStatus.SERVICE_UNAVAILABLE_503); // Change this line!
        }
    }
    finally
    {
        ...
    }
}

Answer 2

503 - Service Unavailable

Apache can't a get a response when proxying the request to the servlet container because, there is no servlet container listening on the host:port specified in the proxy configuration.