Using traefik as a DaemonSet or as a Deployment?

Question

Should I deploy traefik 1.7.x as DaemonSet or as A deployment in GKE (Google K8S)?

Environment Description

Kubernetes clusters with node autoscaler in Google cloud, hosting several production clusters. Clusters can extend up to 90 nodes (minimum is 6 nodes), currently we have traefik pod deployed with 10 replicas in each cluster (we use kustomize to deploy the same manifests in all clusters).

We notice slow response time in the cluster that has 18 nodes ( europe-west1 region), compared to our cluster in australia-southeast1 region, which has 6 nodes. Both clusters has 10 replicas of traefik.

Deployment Specs

traefik.toml:

    [kubernetes]
    # all namespaces!
    namespaces = []

Service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: traefik
  name: traefik-ingress
  namespace: ingress-traefik
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: traefik
  sessionAffinity: None
  type: LoadBalancer
  loadBalancerIP: {{LOAD_BALANCER_IP}}

Deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: traefik
  name: traefik
  namespace: ingress-traefik
spec:
  replicas: 10
  selector:
    matchLabels:
      app: traefik

  template:
    metadata:
      labels:
        app: traefik
    spec:
      containers:
      - args:
        - --configfile=/config/traefik.toml
        image: traefik:1.7.9-alpine

Questions

1. In this scenario (using GKE node autoscaler) what would be the optimal configuration for our clusters? Using Deployment or a DaemonSet for traefik?
2. Does the amount of traefik pods has effect on response time according to the cluster size (node count)?
3. Does routing inside the cluster (hops between pod, service and nodes networks) is easier for traefik when using a DaemonSet (pod for each node) or by using a deployment of several replicas for the whole cluster? (We use K8S namespaces for each of our https service and traefik has its own namespace).

Answer 1

Deploy Traefik using a Deployment or DaemonSet?

It is possible to use Traefik with a Deployment or a DaemonSet object, whereas both options have their own pros and cons:

• The scalability can be much better when using a Deployment, because you will have a Single-Pod-per-Node model when using a DaemonSet, whereas you may need less replicas based on your environment when using a Deployment.

• DaemonSets automatically scale to new nodes, when the nodes join the cluster, whereas Deployment pods are only scheduled on new nodes if required.

• DaemonSets ensure that only one replica of pods run on any single node. Deployments require affinity settings if you want to ensure that two pods don't end up on the same node.

• DaemonSets can be run with the NET_BIND_SERVICE capability, which will allow it to bind to port 80/443/etc on each host. This will allow bypassing the kube-proxy, and reduce traffic hops. Note that this is against the Kubernetes Best Practices Guidelines , and raises the potential for scheduling/scaling issues. Despite potential issues, this remains the choice for most ingress controllers.

There are some significant differences between using Deployments and DaemonSets:

• The Deployment has easier up and down scaling possibilities. It can implement full pod lifecycle and supports rolling updates from Kubernetes 1.2. At least one Pod is needed to run the Deployment.

• The DaemonSet automatically scales to all nodes that meets a specific selector and guarantees to fill nodes one at a time. Rolling updates are fully supported from Kubernetes 1.7 for DaemonSets as well.

You might want to have a look at additional traefik docs