Pod-to-Pod communication is not working on Kubernetes Env

Question

At my Kubernetes environment, I cannot ping pods from other pods. Pods cannot access services, either. Looks like there is a configuration problem. Pod to pod and pod to service communications fail.

I followed the CoreOS + Kubernetes manual steps to install the kubernetes environment (Calico is not installed). https://coreos.com/kubernetes/docs/1.6.1/getting-started.html

Do you have any suggestion how to investigate the problem?

Additional Information ----------------------------

Services created:

system # /home/core/kubernetes/kubectl get services -o wide
NAME            CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE       SELECTOR
kubernetes      10.3.0.1     <none>        443/TCP          35d       <none>
upper-rest      10.3.0.195   <nodes>       5000:32001/TCP   12h       app=upper-rest

Pods:

system # /home/core/kubernetes/kubectl get pods -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP          NODE
upper-rest-4004779333-h7fc2     1/1       Running   0          12h       10.2.58.4   10.201.101.238
upper-rest-4004779333-j444t     1/1       Running   0          12h       10.2.44.2   10.201.101.236

if I ping the service, I get no results

system # ping 10.3.0.195
PING 10.3.0.195 (10.3.0.195) 56(84) bytes of data.
^C
--- 10.3.0.195 ping statistics ---
24 packets transmitted, 0 received, 100% packet loss, time 23555ms

Similarly when I run the ping inside the pod to access a service or other pod's IP, again no reply back.

system # /home/core/kubernetes/kubectl exec upper-rest-4004779333-h7fc2 -- ping 10.3.0.195
PING 10.3.0.195 (10.3.0.195): 56 data bytes
^C

system # /home/core/kubernetes/kubectl exec upper-rest-4004779333-h7fc2 -- ping 10.2.44.2
PING 10.2.44.2 (10.2.44.2): 56 data bytes
^C

Answer 1

that i have no enough reputation to add comment, I think your iptables is old record, maybe you have reset your k8s ever.

I recommanded that you can reset all the cluster, include docker, and clear the iptable and ip link

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

and

ip link delete cni0
ip link delete flannel.1

try with this,

Good luck