nginx ingress setup up on Azure (not using helm) troubleshooting

Question

I need some help trying to achieve this two things here:

1. Setting up my nginx ingress controller on AKS without using helm (I don't want to)
2. Make my ingress to use an already reserved IP address with resource name 'kubernetes-ip'

For the first step I'm following with no luck this documentation: https://kubernetes.github.io/ingress-nginx/deploy/#azure

And I didn't forgot the mandatory.yaml!

my step by step guide:

I have a basic kubernetes cluster with two pods, as follows:

NAME                              READY   STATUS    RESTARTS   AGE
activemq-demo-7b769bcc4-jtsj5     1/1     Running   0          55m
ubuntu-dcb9c6ccb-wkz2w            1/1     Running   0          2d

At this point I want to add my ingress so I can reach the demo activemq using my public ip address abcd

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/cloud-generic.yaml

After doing it I run kubectl get services -n ingress-nginx

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.0.186.143   zz.zz.zzz.zzz   80:32703/TCP,443:30584/TCP   17s

Which is fine! seems to be working right? at this point I should be able to connect to the external ip address to any of those ports but I can't :(

$ telnet zz.zz.zzz.zzz 80
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable

$ telnet zz.zz.zzz.zzz 443
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable

I know that the mandatory.yaml doesn't know nothing about my reserved IP address, but I'm ignoring it because I have a bigger problem, I can't connect.

I also ignore for a minute that I can't connect just to test if I need the actual ingress.yaml running. So I use kubectl apply -f ingress.html . (ingress.yaml contains the following code)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress1
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "*"
spec:
  tls:
  - hosts:
    - amq-test.mydomain.com
    secretName: my-certificate
  rules:
  - host: amq-test.mydomain.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: activemq-demo-service
          servicePort: 8161

After that If I run kubectl get ing I get:

NAME       HOSTS                    ADDRESS         PORTS     AGE
ingress1   amq-test.mydomain.com   zz.zz.zzz.zzz   80, 443   66s

But its the same, I can't connect:

FROM WSL

$ telnet zz.zz.zzz.zzz 80
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable

$ telnet zz.zz.zzz.zzz 443
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable

Honestly I'm not sure what I'm missing, this should be very straight forward with the official documentation, I don't know if I have to enable something else in Azure ...

Thanks for reading, any help will be appreciated.

EDIT 3/7/2020: activemq-demo Deployment & Service @Jean-Philippe Bond

apiVersion: apps/v1
kind: Deployment
metadata:
  name: activemq-demo
  labels:
    app: activemq-demo
    tier: backend
spec:
  revisionHistoryLimit: 1
  replicas: 1
  selector:
    matchLabels:
      app: activemq-demo
  template:
    metadata:
      labels:
        app: activemq-demo
        tier: backend
    spec:
      containers:
      - name: activemq-demo
        image: myproject.azurecr.io/activemq-slim:5.15.9-3
        imagePullPolicy: "Always"
        command: ["/start.sh"]
        args: ["somename"]
        env:
        - name: LANG
          value: "C.UTF-8"
        ports:
        - containerPort: 8161
        - containerPort: 61616
        livenessProbe:
          exec:
            command:
            - /isAlive.sh
            - somename
          initialDelaySeconds: 15
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: activemq-demo-service
  labels:
    tier: controller
spec:
  type: NodePort
  ports:
    - name: http
      protocol: TCP
      port: 8161
      targetPort: 8161
    - name: acceptor
      protocol: TCP
      port: 61616
      targetPort: 61616
  selector:
    app: activemq-demo
    tier: backend

Please note that the only thing I want to access from outside is the HTTP web service that ActiveMQ provides on port 8161 by default

EDIT 3/9/2020: @HelloWorld request

telnet from WSL

$ telnet zz.zz.zzz.zzz 80
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable

$ telnet zz.zz.zzz.zzz 443
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable

telnet from macos

$ telnet zz.zz.zzz.zzz 80
Trying zz.zz.zzz.zzz...
telnet: connect to address zz.zz.zzz.zzz: Operation timed out
telnet: Unable to connect to remote host

$ telnet zz.zz.zzz.zzz 443
Trying zz.zz.zzz.zzz...
telnet: connect to address zz.zz.zzz.zzz: Operation timed out
telnet: Unable to connect to remote host

curl from macos - HTTP

$ curl -v -X GET http://amq-test.mydomain.com
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying zz.zz.zzz.zzz:80...
* TCP_NODELAY set
* Connection failed
* connect to zz.zz.zzz.zzz port 80 failed: Operation timed out
* Failed to connect to amq-test.mydomain.com port 80: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to amq-test.mydomain.com port 80: Operation timed out

curl from macos - HTTPS

$ curl -v -X GET https://amq-test.mydomain.com
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying zz.zz.zzz.zzz:443...
* TCP_NODELAY set
* Connection failed
* connect to zz.zz.zzz.zzz port 443 failed: Operation timed out
* Failed to connect to amq-test.mydomain.com port 443: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to amq-test.mydomain.com port 443: Operation timed out

EDIT 3/10/2020: Starting from scratch (multiple times)

I deleted the whole thing to start "fresh" with no much more luck, but I have noticed some things that hopefully may trigger some thoughts out there...

The basics:

• I already have a Resource Group that I'm using: MyResourceGroup
• I already have a Virtual Network with a Subnet MyVirtualNet
• I already have reserved a Public IP address I want to use with my ingress, It's an Static IP that I want to prevent from changing (or be deleted) until the end of time: ABCD
• I already have my own domain that I have routed to ABCD: amq-test.mydomain.com

My procedure:

1. I'm creating a new Kubernetes Service using the Azure Web Interface, I'm making sure to select my Resource Group as well as my Virtual Network and Subnet
2. As soon I created the base Kubernetes Service I notice that it creates a LoadBalancer with a different Public IP address that I can't control, I assume that its because that IP address will be used as the main entry point for things like kubectl and remote management.
3. With the cluster live I create the basic ActiveMQ image that I shared previously
4. Now, I start with the ingress-nginx and deploy the mandatory.yaml
5. And then I add the Azure Service yaml here , but this time with two modifications to make it use my Public IP address. this modification was extracted from Microsoft documentation

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: MyResourceGroup
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  loadBalancerIP: A.B.C.D
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https

6. With that running without errors, everything is as expect:

$ kubectl get svc -n ingress-nginx -o wide
NAME            TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                      AGE   SELECTOR
ingress-nginx   LoadBalancer   10.0.30.100   A.B.C.D         80:30682/TCP,443:31002/TCP   35m   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx

7. I deploy the final part right? my ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress1
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "*"
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: activemq-demo-service
          servicePort: 8161

This runs without issues and when is up I checked with:

$ kubectl describe ingress
Name:             ingress1
Namespace:        default
Address:          A.B.C.D
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *
        /*   queue-callbacks-service:8161 (10.94.20.14:8161)
Annotations:
  kubernetes.io/ingress.class:                         nginx
  nginx.ingress.kubernetes.io/cors-allow-credentials:  true
  nginx.ingress.kubernetes.io/cors-allow-methods:      *
  nginx.ingress.kubernetes.io/cors-allow-origin:       *
  nginx.ingress.kubernetes.io/enable-cors:             true
  kubectl.kubernetes.io/last-applied-configuration:    {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/cors-allow-credentials":"true","nginx.ingress.kubernetes.io/cors-allow-methods":"*","nginx.ingress.kubernetes.io/cors-allow-origin":"*","nginx.ingress.kubernetes.io/enable-cors":"true"},"name":"ingress1","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"queue-callbacks-service","servicePort":8161},"path":"/callbacks/*"}]}}]}}

Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  35m   nginx-ingress-controller  Ingress default/ingress1
  Normal  UPDATE  34m   nginx-ingress-controller  Ingress default/ingress1

Everything seems to be perfect but I just can't connect to the IP or host using any (80, 443) port

Hope this helps

Just FYI I just tried with helm following this documentation and I got the same result

EDIT 4/15/2020: not done yet

I was working on another project so this one was paused for a moment, I´m returning to this now, unfortunately still not working, I opened a ticket with Microsoft and I'm waiting for it.

However, we noticed that port 80 are being filtered by some firewall or something, we are not sure what is causing this as we have inbound rules on our SG with port 80 and 443 open from * on any protocol

$ nmap -Pn zz.zz.zz.zzz -p 80,443
Starting Nmap 7.80 ( https://nmap.org ) at 2020-04-15 11:01 Pacific SA Standard Time

Nmap scan report for zz.zz.zz.zzz
Host is up.

PORT    STATE    SERVICE
80/tcp  filtered http
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 4.62 seconds

Answer 1

ActiveMQ is accessible via TCP , not HTTP and Kubernetes Ingress were not built to support TCP services. Whit that said, Nginx does support TCP load balancing if you really want to use it, but you'll not be able to use an Ingress rule based on the host like you did since this is reserved for HTTP/HTTPS . Your best bet would probably be to use the Azure L4 Load balancer directly instead of going through the ingress controller.

If you want to use Nginx , you'll need to modify the yaml in mandatory.yaml to expose the ActiveMQ port on the Nginx deployment.

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  9000: "default/activemq-demo-service:8161"

You'll also need to add the tcp-services port on the Service resource. Ex :

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
    - name: proxied-tcp-9000
      port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

Here is the documentation for TCP/UDP support in the Nginx Ingress.

Answer 2

前段时间我写了一个可能对你有用的故事： https : //medium.com/cooking-with-azure/tcp-load-balancing-with-ingress-in-aks-702ac93f2246