Unable to access to service from kubernetes master node

Question

[root@kubemaster ~]# kubectl get pods -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
pod1deployment-c8b9c74cb-hkxmq   1/1     Running   0          12s   192.168.90.1   kubeworker1   <none>           <none>

[root@kubemaster ~]# kubectl logs pod1deployment-c8b9c74cb-hkxmq
2020/05/16 23:29:56 Server listening on port 8080

[root@kubemaster ~]# kubectl get service -o wide
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP   13m   <none>
pod1service   ClusterIP   10.101.174.159   <none>        80/TCP    16s   creator=sai

Curl on master node:

[root@kubemaster ~]# curl -m 2 -v -s http://10.101.174.159:80
* About to connect() to 10.101.174.159 port 80 (#0)
*   Trying 10.101.174.159...
* Connection timed out after 2001 milliseconds
* Closing connection 0

Curl on worker node 1 is sucessfull for cluster IP ( this is the node where pod is running )

[root@kubemaster ~]# ssh kubeworker1 curl -m 2 -v -s http://10.101.174.159:80
Hello, world!
Version: 1.0.0
Hostname: pod1deployment-c8b9c74cb-hkxmq

Curl fails on other worker node as well:

[root@kubemaster ~]# ssh kubeworker2 curl -m 2 -v -s http://10.101.174.159:80
* About to connect() to 10.101.174.159 port 80 (#0)
*   Trying 10.101.174.159...
* Connection timed out after 2001 milliseconds
* Closing connection 0

Answer 1

I was facing the same issue so this is what I did and it worked:

Brief: I am running 2 VMs for a 2 Node cluster. 1 Master Node and 1 Worker Node. A Deployment is running on the worker node. I wanted to curl from the master node so that I can get response from my application running inside a pod on the worker node. For that I deployed a service on the worker node which then exposed those set of pods inside the cluster.

Issue: After deploying the service and doing Kubectl get service , it provided me with ClusterIP of that service and a port (BTW I used NodePort instead of Cluster IP when writing the service.yaml). But when curling on that IP address and port it was just hanging and then after sometime giving timeout.

Solution: Then I tried to look at the hierarchy. First I need to contact the Node on which service is located then on the port given by the NodePort (ie The one between 30000-32767) so first I did Kubectl get nodes -o wide to get the Internal IP address of the required Node (mine was 10.0.1.4) and then I did kubectl get service -o wide to get the port (the one between 30000-32767) and curled it. So my curl command was -> curl http://10.0.1.4:30669 and I was able to get the output.

Answer 2

First of all, you should always be using Service DNS instead of Cluster/dynamic IPs to access the application deployed. The service DNS would be < service-name >.< service-namespace >.svc.cluster.local , cluster.local is the default Kube.netes cluster name, if not changed otherwise.

Now coming to the service accessibility, it may be DNS issues. What you can do is try to check the kube-dns pod logs in kube-system namespace. Also, try to curl from a standalone pod. If that's working.

kubectl run --generator=run-pod/v1 bastion --image=busybox

kubectl exec -it bastion bash

curl -vvv pod1service.default.svc.cluster.local

If not the further questions would be, where is the cluster and how it was created?