Kubernetes ingress服务NodePort只监听tcp6 / ipv6而不是tcp / ipv4

Question

I have setup ingress-nginx using helm through helm install --name x2f1 stable/nginx-ingress --namespace ingress-nginx and service:

apiVersion: v1
kind: Service
metadata:
  name: x2f1-ingress-nginx-svc
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 30080
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      nodePort: 30443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

running svc and po's:

[ottuser@ottorc01 ~]$ kubectl get svc,po -n ingress-nginx
NAME                                         TYPE           CLUSTER-IP        EXTERNAL-IP   PORT(S)                      AGE
service/x2f1-ingress-nginx-svc               NodePort       192.168.34.116    <none>        80:30080/TCP,443:30443/TCP   2d18h
service/x2f1-nginx-ingress-controller        LoadBalancer   192.168.188.188   <pending>     80:32427/TCP,443:31726/TCP   2d18h
service/x2f1-nginx-ingress-default-backend   ClusterIP      192.168.156.175   <none>        80/TCP                       2d18h

NAME                                                      READY   STATUS    RESTARTS   AGE
pod/x2f1-nginx-ingress-controller-cd5fbd447-c4fqm         1/1     Running   0          2d18h
pod/x2f1-nginx-ingress-default-backend-67f8db4966-nlgdd   1/1     Running   0          2d18h

after that my nodePort: 30080 is only available against tcp6, due to this, im facing connection refused when try to access from other vm.

[ottuser@ottorc01 ~]$ netstat -tln | grep '30080'
tcp6       3      0 :::30080                :::*                    LISTEN

[ottuser@ottwrk02 ~]$ netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:6443          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN
tcp        0      0 10.18.0.10:2379         0.0.0.0:*               LISTEN
tcp        0      0 10.18.0.10:2380         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:33372         0.0.0.0:*               LISTEN
tcp6       0      0 :::10250                :::*                    LISTEN
tcp6       0      0 :::30443                :::*                    LISTEN
tcp6       0      0 :::32427                :::*                    LISTEN
tcp6       0      0 :::31726                :::*                    LISTEN
tcp6       0      0 :::10256                :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::30462                :::*                    LISTEN
tcp6       0      0 :::30080                :::*                    LISTEN

Logs from pod/x2f1-nginx-ingress-controller-cd5fbd447-c4fqm :

[ottuser@ottorc01 ~]$ kubectl logs pod/x2f1-nginx-ingress-controller-cd5fbd447-c4fqm -n ingress-nginx --tail 50
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.24.1
  Build:      git-ce418168f
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

I0621 11:48:26.952213       6 flags.go:185] Watching for Ingress class: nginx
W0621 11:48:26.952772       6 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.10
W0621 11:48:26.961458       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0621 11:48:26.961913       6 main.go:205] Creating API client for https://192.168.0.1:443
I0621 11:48:26.980673       6 main.go:249] Running in Kubernetes cluster version v1.14 (v1.14.1) - git (clean) commit b7394102d6ef778017f2ca4046abbaa23b88c290 - platform linux/amd64
I0621 11:48:26.986341       6 main.go:102] Validated ingress-nginx/x2f1-nginx-ingress-default-backend as the default backend.
I0621 11:48:27.339581       6 main.go:124] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
I0621 11:48:27.384666       6 nginx.go:265] Starting NGINX Ingress controller
I0621 11:48:27.403396       6 event.go:209] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"x2f1-nginx-ingress-controller", UID:"89b4caf0-941a-11e9-a0fb-005056010a71", APIVersion:"v1", ResourceVersion:"1347806", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/x2f1-nginx-ingress-controller
I0621 11:48:28.585472       6 nginx.go:311] Starting NGINX process
I0621 11:48:28.585630       6 leaderelection.go:217] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
W0621 11:48:28.586778       6 controller.go:373] Service "ingress-nginx/x2f1-nginx-ingress-default-backend" does not have any active Endpoint
I0621 11:48:28.586878       6 controller.go:170] Configuration changes detected, backend reload required.
I0621 11:48:28.592786       6 status.go:86] new leader elected: x2f1-ngin-nginx-ingress-controller-567f495994-hmcqq
I0621 11:48:28.761600       6 controller.go:188] Backend successfully reloaded.
I0621 11:48:28.761677       6 controller.go:202] Initial sync, sleeping for 1 second.
[21/Jun/2019:11:48:29 +0000]TCP200000.001
W0621 11:48:32.444623       6 controller.go:373] Service "ingress-nginx/x2f1-nginx-ingress-default-backend" does not have any active Endpoint
[21/Jun/2019:11:48:35 +0000]TCP200000.000
I0621 11:49:05.793313       6 status.go:86] new leader elected: x2f1-nginx-ingress-controller-cd5fbd447-c4fqm
I0621 11:49:05.793331       6 leaderelection.go:227] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0621 11:53:08.579333       6 controller.go:170] Configuration changes detected, backend reload required.
I0621 11:53:08.579639       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ott", Name:"hie-01-hie", UID:"32678e25-941b-11e9-a0fb-005056010a71", APIVersion:"extensions/v1beta1", ResourceVersion:"1348532", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress ott/hie-01-hie
I0621 11:53:08.764204       6 controller.go:188] Backend successfully reloaded.
[21/Jun/2019:11:53:08 +0000]TCP200000.000
I0621 11:54:05.812798       6 status.go:295] updating Ingress ott/hie-01-hie status from [] to [{ }]

[ottuser@ottorc01 ~]$ sudo ss -l -t -p | grep 30080
LISTEN     3      128       :::30080                   :::*                     users:(("kube-proxy",pid=29346,fd=15))

Is there any way to debug it in further depth or add that port to tcp/ipv4. If still something unclear from my side let me know. Thanks in advance.

Answer 1

It's not a problem of the tcp6.

On most modern Linux distros, including Container Linux, listening on tcp6 will also imply tcp4.

The issue itself is with your x2f1-ingress-nginx-svc service and specifically with selectors, which do not match with any pod

selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

If you will do

kubectl get ep -n ingress-nginx

you will see that there's no endpoints for that service

NAME                                 ENDPOINTS                        AGE
x2f1-ingress-nginx-svc               <none>                           13m

Now the question is what do you want to expose with this service?

For instance, if you will be exposing x2f1-nginx-ingress-controller (even though helm already created appropriate service), your yaml should be like:

apiVersion: v1
kind: Service
metadata:
  name: x2f1-ingress-nginx-svc
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 30080
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      nodePort: 30443
  selector:
     app: nginx-ingress
     component: controller