Camunda External client: TASK/CLIENT-02002 Exception while establishing connection for request

Question

We started with the implementation of a camunda based workflow solution. In the moment the setup is like this:

1. A spring boot application with an embedded camunda BPM (via camunda-bpm-spring-boot-starter-rest and camunda-bpm-spring-boot-starter-webapp)
2. A spring boot application with an external task client (via camunda-external-task-client)

Everything is working fine so far. Our workflow is running and the external client is doing his job... But after a while (when there is nothing to do for the external client) is see an exception in the log of the external task client:

15:49:09.692 [E] [TopicSubscripti] client.logError:70 - TASK/CLIENT-03001 Exception while fetch and lock task.
org.camunda.bpm.client.impl.EngineClientException: TASK/CLIENT-02002 Exception while establishing connection for request 'POST http://localhost:8080/enrichmentservice/api/rest/1.0/rest/external-task/fetchAndLock HTTP/1.1'
at org.camunda.bpm.client.impl.EngineClientLogger.exceptionWhileEstablishingConnection(EngineClientLogger.java:36)
at org.camunda.bpm.client.impl.RequestExecutor.executeRequest(RequestExecutor.java:101)
at org.camunda.bpm.client.impl.RequestExecutor.postRequest(RequestExecutor.java:74)
at org.camunda.bpm.client.impl.EngineClient.fetchAndLock(EngineClient.java:72)
at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.fetchAndLock(TopicSubscriptionManager.java:135)
at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.acquire(TopicSubscriptionManager.java:101)
at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.run(TopicSubscriptionManager.java:87)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: localhost:8080 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
at org.camunda.bpm.client.impl.RequestExecutor.executeRequest(RequestExecutor.java:88)
at org.camunda.bpm.client.impl.RequestExecutor.postRequest(RequestExecutor.java:74)
at org.camunda.bpm.client.impl.EngineClient.fetchAndLock(EngineClient.java:72)
at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.fetchAndLock(TopicSubscriptionManager.java:135)

What could be the reason for this? Maybe Configuration error in server or client? One remark: The execution of the external task is slow (like 10-30 seconds)

Update: I created a complete example: https://c.gmx.net/@505442085592110443/BIItJGdwTcuWwk7_XqNOXw

To create the error scenario you have to:

1. Start the ExampleApplication inside the spring-boot project
2. Start the ExternalClientApp inside the Spring-Boot-Client project
3. Wait a few minutes

the log output of the external client should look like this:

Subscribe client for: approveLoan
Subscription done
Subscribe client for: waitTask
Subscription done
pojo before: ObjectValue [value=ExamplePojo [num=123, textVal=some text], isDeserialized=true, serializationDataFormat=application/x-java-serialized-object, objectTypeName=org.camunda.bpm.example.tasks.ExamplePojo, serializedValue=156 chars, isTransient=false]
pojo changed: ObjectValue [value=ExamplePojo [num=123, textVal=external changed], isDeserialized=true, serializationDataFormat=application/x-java-serialized-object, objectTypeName=org.camunda.bpm.example.tasks.ExamplePojo, serializedValue=156 chars, isTransient=false]
The External Task 28 has been completed!
The External Task 32 has been completed! (done = false)
The External Task 39 has been completed! (done = false)
The External Task 46 has been completed! (done = true)
149038 [TopicSubscriptionManager] ERROR org.camunda.bpm.client - TASK/CLIENT-03001 Exception while fetch and lock task.
org.camunda.bpm.client.impl.EngineClientException: TASK/CLIENT-02002 Exception while establishing connection for request 'POST http://localhost:8080/rest/external-task/fetchAndLock HTTP/1.1'
at org.camunda.bpm.client.impl.EngineClientLogger.exceptionWhileEstablishingConnection(EngineClientLogger.java:36)
at org.camunda.bpm.client.impl.RequestExecutor.executeRequest(RequestExecutor.java:101)
at org.camunda.bpm.client.impl.RequestExecutor.postRequest(RequestExecutor.java:74)
at org.camunda.bpm.client.impl.EngineClient.fetchAndLock(EngineClient.java:72)
at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.fetchAndLock(TopicSubscriptionManager.java:135)
at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.acquire(TopicSubscriptionManager.java:101)
at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.run(TopicSubscriptionManager.java:87)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: localhost:8080 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
at org.camunda.bpm.client.impl.RequestExecutor.executeRequest(RequestExecutor.java:88)
... 6 more

Answer 1

After testing it on different machines/jdk/OS it looks like the problem is only on windows 7 machines.

i think the error reason is described here: Apache HttpClient Interim Error: NoHttpResponseException

so the server is killing the http session because it has not been used for a longer time!

my workaround is to configure a different backup strategy in the client and so far i have no errors anymore:

tldr: i think it is a bug in the external client that only happens on Win7 but i found a workaround:

    ExternalTaskClient client = ExternalTaskClient.create().baseUrl(baseUrl).backoffStrategy(new ExponentialBackoffStrategy(500L, 2, 30000L)).
            build();

PS i reported it as a bug: https://app.camunda.com/jira/browse/CAM-10526