Hystrix: Apache http client requests are not interrupted

Question

I have an apache HTTP client defined as follows:

private static HttpClient httpClient = null;
HttpParams httpParams = new BasicHttpParams();
httpParams.setParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, Boolean.TRUE);
httpParams.setParameter(CoreProtocolPNames.USER_AGENT, "ABC");

HttpConnectionParams.setStaleCheckingEnabled(httpParams, Boolean.TRUE);

SSLSocketFactory sf = SSLSocketFactory.getSocketFactory();

SchemeRegistry schemeRegistry = new SchemeRegistry();
schemeRegistry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
schemeRegistry.register(new Scheme("https", 443, sf));

//Initialize the http connection pooling
PoolingClientConnectionManager connectionManager = new PoolingClientConnectionManager(schemeRegistry);

// Initialize the connection parameters for performance tuning
connectionManager.setMaxTotal(12);
connectionManager.setDefaultMaxPerRoute(10);

httpClient = new DefaultHttpClient(connectionManager, httpParams);

I have a hystrix command play and have the following properties enabled:

hystrix.command.play.execution.isolation.thread.timeoutInMilliseconds=1
hystrix.command.play.execution.isolation.thread.interruptOnTimeout=true

The command itself is defined as follows:

    @HystrixCommand(groupKey="play_group",commandKey="play")
    public String process(String request) throws UnsupportedOperationException, IOException, InterruptedException {
        System.out.println("Before -  process method : " + request);
        callHttpClient(request);
        System.out.println("After -  process method" + request);
        return "";
    }

    private void callHttpClient(String request) throws ClientProtocolException, IOException, InterruptedException {
        HttpGet get = new HttpGet("http://www.google.co.in");
        HttpResponse response = httpClient.execute(get);
        System.out.println("Response:" + response);
    }

I now try executing the command 5 times in a loop:

    public static void main(String[] args) throws UnsupportedOperationException, IOException, InterruptedException {
        ApplicationContext ctx = new ClassPathXmlApplicationContext("applicationContextTest.xml");
        HystrixPlayground obj = ctx.getBean(HystrixPlayground.class);

        long t1 = System.currentTimeMillis();
        for (int i = 0; i < 5; i++) {
            try{
                System.out.println(obj.process("test" + i));
            } catch(Exception ex) {
                System.out.println(ex);
            }
            long t2 = System.currentTimeMillis();
            System.out.println("Time(ms) : ---->" + (t2 - t1));

The timeout is set to 1 milli second, so the process method throws a HystrixRunTimeException. However, the http request continues to execute and prints the "After - process method" string.

I have seen this behavior consistently for http client requests only. If the http request is replaced by anything else like a thread sleep or a very big for loop, the hystrix thread gets interrupted as expected.

Does anyone have any insight into why this may be happening?

Answer 1

The reason is that interrupting a thread in Java does not "force stop" it. Instead, invoking Thread.interrupt() just sets a flag which can, but does not have to be interpreted by the running thread. See more here: What does java.lang.Thread.interrupt() do?

Apache HTTP client does not interpret this flag (as a socket read operation by the JRE cannot be interrupted). Thus, the HTTP request is not canceled and just completes.

Answer 2

It seems that by the past some IO operations, on some plaforms(solaris) could be interrupted using Thread.interrupt() . Now that seems not not possible/specified anymore for java.io , but still possible with java.nio. (see Java bug 4385444 and Java bug 7188233 for more details)

So even if with some platforms/versions it seems still possible to interrupt a blocking IO with Thread.interrupt() , this seems clearly not recommended :

• https://www.oreilly.com/library/view/java-threads-second/1565924185/ch04s05.html#ch04-8-fm2xml
• https://wiki.sei.cmu.edu/confluence/display/java/THI04-J.+Ensure+that+threads+performing+blocking+operations+can+be+terminated

So there is nothing wrong in httpclient . With Hystrix you should abort() the request on Hystrix timeout. If you are using fallback, you can call it in getFallback() , if not you can call it after you get the result. Eg :

try {
    hystrixCommand.execute();
} catch (HystrixRuntimeException e) {
    switch (e.getFailureType()) {
    case TIMEOUT:
       // supposing you add a getHttpRequest getter on your command.
       hystrixCommand.getHttpRequest().abort();
    }
}