kubernetes: cannot access service on VirtualBox cluster via ClusterIP / NodePort

Question

I created a 3-node kubernetes cluster (1 master + 2 workers) on VirtualBox using the instructions here . I am using Flannel for the overlay network.

I set sysctl -w net.bridge.bridge-nf-call-iptables=1 and sysctl -w net.bridge.bridge-nf-call-ip6tables=1 on the master during installation. I hadn't set them on the workers at that time, but I set them later and rebooted both nodes.

1. I have a trivial web app written in Go, listening on port 8080. I created a pod + replication controller thus:

    kubectl run foo --image=<...> --port=8080 --generator=run/v1 

   I am able to access my service using the POD IP and port 8080.

2. I also created a ClusterIP service.

    kubectl expose rc foo --name=foo-http --port=8081 --target-port=8080 # ClusterIP service # kubectl get svc foo-http NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE foo-http ClusterIP 10.106.88.24 <none> 8081/TCP 14m 

   When I run this from any cluster node, it hangs:

    curl http://10.106.88.24:8081 # that's the ClusterIP 

   By running strace, I can see that curl initiates a non-blocking connect , and spins in a loop on poll , without the socket ever becoming ready for a read - so the connection doesn't go through.

3. If I create a NodePort service instead, I simply get connection refused.

    # cat svc_nodeport.json apiVersion: v1 kind: Service metadata: name: foo-http spec: type: NodePort ports: - port: 8081 targetPort: 8080 nodePort: 31123 selector: app: foo # kubectl create -f svc_nodeport.json # kubectl get svc foo-http NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE foo-http NodePort 10.104.78.88 <none> 8081:31123/TCP 7m 

   When I try connecting via port 31123:

    # curl http://<node-ip>:31123 # Tried on master and both workers curl: (7) Failed connect to <node-ip>:31123; Connection refused 

How do debug this?

Answer 1

I think your service manifest selector isn't selecting the pod. Try to describe the pod and check for the labels. Kubectl describe pod Then change your service manifest according to your pod labels.

Best practice is to use declarative ways by writing the deployment or pod manifest, instead of using the imperative commands of kubectl, such as run or create.

Answer 2

check first if something wrong :

kubectl describe service foo-http

and try other network cluster (for example: remove current network and deploy weave network)

 kubectl get -n kube-system daemonset | grep -i Flannel |cut -f1 -d ' '| kubectl delete daemonset -n kube-system 
 kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

and for testing change "port" and "targetPort" to 8080 as you mentioned (or remove "targetPort") , like :

ports:
      - port: 8080
        targetPort: 8080
        nodePort: 31123

then

kubectl delete service foo-http

and create your service again Ex.

kubectl create -f UrServiceScript.yaml