Odd unknown host exception

Question

I have a server that is experiencing some wierd exceptions. The server is calling a backend service and during "normal" operation this works without fault, however if that service goes down for a short timespan my server completely dies. After the backend service has restarted I am getting unknown host exceptions for all my http calls. If I restart my server it works again, but it seems odd that I even get the error in the first place.

An example of a call I make:

        Client client = ClientBuilder.newClient();

        response = client.target(uri)
                .request(MediaType.APPLICATION_JSON_TYPE)
                .header("XApiKey", catalogApiKey)
                .get();

and the stacktrace is rather standard:

java.net.UnknownHostException.
      at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
      at java.net.InetAddress.getAllByName(InetAddress.java:1192)
      at java.net.InetAddress.getAllByName(InetAddress.java:1126)
      at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
      at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:259)
      at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:159)
      at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
      at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
      at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
      at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
      at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:436)
      at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:159)

if it is relevant I am running on a wildfly server.

What could cause this kind of behaviour?

Answer 1

If you look at the source of InetAddress (eg http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/net/InetAddress.java ) everywhere that UnknownHostException is instantiated it is instantiated with either a host string or a clearly non-empty string as the constructor parameter.

However, the exception in the stacktrace appears to have an empty message . So I infer that the code is doing a lookup on a hostname that is an empty string.

I would add some code to print the value of uri when this problem occurs. I suspect that is (sporadically) bogus in some interesting way.

Answer 2

Java caches DNS entries. The default is "forever" as defined in $JAVA_HOME/jre/lib/security/java.security under the entry networkaddress.cache.ttl .

I would encourage you to consider reducing the "positive" cache to a small amount, perhaps 10 seconds or so. You can modify the java.security file or have code like:

java.security.Security.setProperty("networkaddress.cache.ttl", "10");

In your code. There is also a "negative" cache that caches entries that have failed. It may take some experimentation in your environment with the positive and negative caches to find the right setting.