Linkerd traffic split with Nginx Ingress Controller

Question

I have deployed a Linkerd Service mesh and my Kubernetes cluster is configured with the Nginx ingress controller as a DaemonSet and all the ingresses are working fine also the Linkerd. Recently, I have added a traffic split functionality to run my blue/green setup I can reach through to these services with separate ingress resources. I have created an apex-web service as described here . If I reached you this service internally it perfectly working. I have created another ingress resources and I'm not able to test the blue/green functionality outside of my cluster. I'd like to mention that I have meshed (injected the Linkerd proxy) to all my Nginx pods but it is returning " 503 Service Temporarily Unavailable " message from the Nginx.

I went through the documentation and I have created ingress following this , I can confirm that below annotations were added to the ingress resources.

annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
  grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;

but still no luck with the out side of the cluster.

I'm testing with the given emojivoto app and all the traffic split and the apex-web services are in this training repository.

I'm not quite sure what went wrong and how to fix this outside from the cluster. I'd really appreciate if anyone assist me to fix this Linkerd, Blue/Green issue.

Answer 1

tl;dr: The nginx ingress requires a Service resource to have an Endpoint resource in order to be considered a valid destination for traffic. The architecture in the repo creates three Service resources, one of which acts as an apex and has no Endpoint resources because it has no selectors, so the nginx ingress won't send traffic to it, and the leaf services will not get traffic as a result.

The example in the repo follows the SMI Spec by defining a single apex service and two leaf services. The web-apex service does not have any endpoints, so nginx will not send traffic to it.

According to the SMI Spec services can be self-referential , which means that a service can be both an apex and a leaf service, so to use the nginx ingress with this example, you can modify the TrafficSplit definition to change the spec.service value from web-apex to web-svc :

apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: web-svc-ts
  namespace: emojivoto
spec:
  # The root service that clients use to connect to the destination application.
  service: web-svc
  # Services inside the namespace with their own selectors, endpoints and configuration.
  backends:
  - service: web-svc
    # Identical to resources, 1 = 1000m
    weight: 500m
  - service: web-svc-2
    weight: 500m

Answer 2

I have raised this question in the Linkerd Slack channel and got this fixed with the wonderful support from the community. Seems Nginx doesn't like the service which doesn't have an endpoint. My configuration was correct and asked to change the service pointed in the traffic split to a service with an endpoint and it fixed the issue.

In a nutshell, my traffic split was configured with web-svc and web-svc-2 services. I have changed the traffic split spec.service to the same web-svc and it worked

Here is the traffic split configuration after the update.

apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: web-svc-ts
  namespace: emojivoto
spec:
  # The root service that clients use to connect to the destination application.
  service: web-svc
  # Services inside the namespace with their own selectors, endpoints and configuration.
  backends:
  - service: web-svc
    # Identical to resources, 1 = 1000m
    weight: 500m
  - service: web-svc-2
    weight: 500m

Kudos to the Linkerd team who supported me to fix this issue. It is working like a charm.