ElasticSearch java.net.NoRouteToHostException in docker

Question

[2015-10-11 13:08:26,587][WARN ][transport.netty          ] [Joseph] exception caught on transport layer [[id: 0x7e9f652b]], closing connection
java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

I get this exception when launching the elasticsearch in docker (Actually I only have this problem in CentOS7 docker host)

First, my dockefile exposes the UDP ports.

EXPOSE 9200 9300/udp 9301/udp 9302/udp 9303/udp 9304/udp 9305/udp

When I start the docker container, I opened these ports via -p 9200:9200 -p 9300:9300/udp -p 9301:9301/udp -p 9302:9302/udp -p 9303:9303/udp -p 9304:9304/udp -p 9305:9305/udp

Within docker ps , I do see these ports are opened as 0.0.0.0:9300-9305->9300-9305/udp

And here is some lines of my elasticsearch.yml

cluster.name: changsha
discovery.zen.ping.unicast.hosts: [ "10.0.5.241" ]
network.publish_host: 10.0.5.241

10.0.5.241 is my docker host's IP address. Please what is wrong here? it succeeded in CentOS6 host, but failes on this CentOS7 host.

UPDATE

Following this answer , I get the following result from tcpdump -p -nn icmp .

09:26:53.277117 IP 10.0.5.241 > 172.17.0.8: ICMP host 10.0.5.241 unreachable - admin prohibited, length 68
09:26:53.277494 IP 10.0.5.241 > 172.17.0.8: ICMP host 10.0.5.241 unreachable - admin prohibited, length 68
09:26:53.277822 IP 10.0.5.241 > 172.17.0.8: ICMP host 10.0.5.241 unreachable - admin prohibited, length 68
09:26:53.278043 IP 10.0.5.241 > 172.17.0.8: ICMP host 10.0.5.241 unreachable - admin prohibited, length 68
09:26:54.277753 IP 10.0.5.241 > 172.17.0.8: ICMP host 10.0.5.241 unreachable - admin prohibited, length 68
09:27:04.280703 IP 10.0.5.241 > 172.17.0.8: ICMP host 10.0.5.241 unreachable - admin prohibited, length 68

Answer 1

First, find out the docker interface ip address

# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.42.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 56:84:7a:fe:97:99  txqueuelen 0  (Ethernet)
        RX packets 115761  bytes 12605533 (12.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55687  bytes 22647938 (21.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Then add all of the docker IP addresses into whitelist

firewall-cmd --permanent --zone=trusted --add-source=172.17.0.0/16
firewall-cmd --reload

Problem solved

Answer 2

If someone come across the issue in centos 7.4, it`s because of the conflict between docker service and firewalld service.

you can solve by disable firewalld and then restart docker service.

please refer https://sanenthusiast.com/docker-and-firewalld-mess-in-centos-7/