AWS Kubernetes Exposed Service Timeout Error

Question

I got an issue when I try to access an exposed kubernetes service through browser. Below is my Environment.

created two ubuntu EC2 instances(with all ports open in security group) and installed all kubernetes related tools like kubectl, kubeadm, docker, calico network.

created nginx pod, scaled it to 3 and exposed it with type . 类型将其公开。 When I curl from master or worker node to the exposed nginx it works fine(with public or private ip). But it does not work if i curl from outside. The request is timed out. I tried to delete service and expose it again with NodePort. But still I could not access from outside. I ensured the security group allows ingress. Is there a way to debug why it cannot be accessed from outside or I am missing something

I am not running cloud controller manager but kube-controller-manager . Will this be an issue.?

below is the output of all kubernetes components

ubuntu@ip-172-31-29-98:~$ kubectl get all --all-namespaces
NAMESPACE     NAME                                           READY     STATUS    RESTARTS   AGE
default       pod/nginx-6f858d4d45-2wtlh                     1/1       Running   0          51m
default       pod/nginx-6f858d4d45-5dkws                     1/1       Running   0          51m
default       pod/nginx-6f858d4d45-h9cwg                     1/1       Running   0          51m
kube-system   pod/calico-etcd-82xkv                          1/1       Running   1          18h
kube-system   pod/calico-kube-controllers-74b888b647-prr2q   1/1       Running   1          18h
kube-system   pod/calico-node-kbckk                          2/2       Running   4          17h
kube-system   pod/calico-node-n5zhr                          2/2       Running   3          18h
kube-system   pod/coredns-78fcdf6894-qjhlq                   1/1       Running   1          18h
kube-system   pod/coredns-78fcdf6894-sm7c9                   1/1       Running   1          18h
kube-system   pod/etcd-ip-172-31-29-98                       1/1       Running   1          18h
kube-system   pod/kube-apiserver-ip-172-31-29-98             1/1       Running   1          18h
kube-system   pod/kube-controller-manager-ip-172-31-29-98    1/1       Running   1          18h
kube-system   pod/kube-proxy-jxg88                           1/1       Running   1          18h
kube-system   pod/kube-proxy-knx59                           1/1       Running   1          17h
kube-system   pod/kube-scheduler-ip-172-31-29-98             1/1       Running   1          18h

NAMESPACE     NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes    ClusterIP      10.96.0.1       <none>        443/TCP         18h
default       service/nginx         LoadBalancer   10.99.144.149   <pending>     80:31808/TCP    45m
kube-system   service/calico-etcd   ClusterIP      10.96.232.136   <none>        6666/TCP        18h
kube-system   service/kube-dns      ClusterIP      10.96.0.10      <none>        53/UDP,53/TCP   18h

NAMESPACE     NAME                         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
kube-system   daemonset.apps/calico-etcd   1         1         1         1            1           node-role.kubernetes.io/master=   18h
kube-system   daemonset.apps/calico-node   2         2         2         2            2           <none>                            18h
kube-system   daemonset.apps/kube-proxy    2         2         2         2            2           beta.kubernetes.io/arch=amd64     18h

NAMESPACE     NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
default       deployment.apps/nginx                      3         3         3            3           51m
kube-system   deployment.apps/calico-kube-controllers    1         1         1            1           18h
kube-system   deployment.apps/calico-policy-controller   0         0         0            0           18h
kube-system   deployment.apps/coredns                    2         2         2            2           18h

NAMESPACE     NAME                                                  DESIRED   CURRENT   READY     AGE
default       replicaset.apps/nginx-6f858d4d45                      3         3         3         51m
kube-system   replicaset.apps/calico-kube-controllers-74b888b647    1         1         1         18h
kube-system   replicaset.apps/calico-policy-controller-55b469c8fd   0         0         0         18h
kube-system   replicaset.apps/coredns-78fcdf6894                    2         2         2         18h

Edit 1: Tried to do the same in GCloud . It is same in GCloud too. The nginx Service is accessible through private/public ip inside the nodes. But when I curl from outside, it does not work. I spin up a simply python server in one of the node and I am able to access the python server from outside. But only the services exposed through kubernetes are not curl-able from outside.

I think I am missing some fundamental understanding about kubernetes networking(especially in cloud). Can I get any help from experts?

Answer 1

The internet firewall on my network was blocking the access to specific ports. There was no issues with the Cloud Provider Firewall or Instance Firewall.

Answer 2

How did you create your cluster? kubeadm ? You need some custom configs for your cluster to run with AWS. For example your kube-controller-manager and kube-apiserver need to have the option --cloud-provider=aws . Same for all your kubelets.

I'd recommend using kops if you don't want to deal with this.