Kubernetes - Too many open files

Question

I'm trying to evaluate the performance of one of my go server running inside the pod. However, receiving an error saying too many open files. Is there any way to set the ulimit in kube.netes?

ubuntu@ip-10-0-1-217:~/ppu$ kubectl exec -it go-ppu-7b4b679bf5-44rf7 -- /bin/sh -c 'ulimit -a'
core file size (blocks)         (-c) unlimited
data seg size (kb)              (-d) unlimited
scheduling priority             (-e) 0
file size (blocks)              (-f) unlimited
pending signals                 (-i) 15473
max locked memory (kb)          (-l) 64
max memory size (kb)            (-m) unlimited
open files                      (-n) 1048576
POSIX message queues (bytes)    (-q) 819200
real-time priority              (-r) 0
stack size (kb)                 (-s) 8192
cpu time (seconds)              (-t) unlimited
max user processes              (-u) unlimited
virtual memory (kb)             (-v) unlimited
file locks                      (-x) unlimited

Deployment file.

---
apiVersion: apps/v1
kind: Deployment                 # Type of Kubernetes resource
metadata:
  name: go-ppu           # Name of the Kubernetes resource
spec:
  replicas: 1                    # Number of pods to run at any given time  
  selector:
    matchLabels:
      app: go-ppu         # This deployment applies to any Pods matching the specified label
  template:                      # This deployment will create a set of pods using the configurations in this template
    metadata:
      labels:                    # The labels that will be applied to all of the pods in this deployment
        app: go-ppu  
    spec:                        # Spec for the container which will run in the Pod
      containers:
      - name: go-ppu 
        image: ppu_test:latest
        imagePullPolicy: Never
        ports:
          - containerPort: 8081  # Should match the port number that the Go application listens on
        livenessProbe:           # To check t$(minikube docker-env)he health of the Pod
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 35
          periodSeconds: 30
          timeoutSeconds: 20
        readinessProbe:          # To check if the Pod is ready to serve traffic or not
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 35
          timeoutSeconds: 20    

Pods info:

ubuntu@ip-10-0-1-217:~/ppu$ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
go-ppu-7b4b679bf5-44rf7        1/1     Running   0          18h

ubuntu@ip-10-0-1-217:~/ppu$ kubectl get services
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP                                                               PORT(S)          AGE
kubernetes    ClusterIP      100.64.0.1      <none>                                                                    443/TCP          19h
ppu-service   LoadBalancer   100.64.171.12   74d35bb2a5f30ca13877-1351038893.us-east-1.elb.amazonaws.com   8081:32623/TCP   18h

When I used locust to test the performance of the server receiving the following error.

# fails Method  Name    Type
3472    POST    /supplyInkHistory   ConnectionError(MaxRetryError("HTTPConnectionPool(host='74d35bb2a5f30ca13877-1351038893.us-east-1.elb.amazonaws.com', port=8081): Max retries exceeded with url: /supplyInkHistory (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x....>: Failed to establish a new connection: [Errno 24] Too many open files',))",),)

Answer 1

May you have a look at https://kube.netes.io/docs/tasks/administer-cluster/sysctl-cluster/ You but you need enable few features to make it work.

  securityContext:
sysctls:
- name: fs.file-max
  value: "YOUR VALUE HERE"

Answer 2

There was a few cases regarding setting --ulimit argument, you can find them here or check this article . This resource limit can be set by Docker during the container startup. As you add tag google-kube.netes-engine answer will be related to GKE environment, however on other cloud it could work similar.

If you would like to set unlimit for open files you can modify configuration file /etc/security/limits.conf . However, please not it will not persist across reboots.

Second option would be edit /etc/init/docker.conf and restart docker service. As default it have a few limits like nofile or nproc , you can add it here.

Another option could be to use instance template . Instance template would include a start-up script that set the required limit. After that, you would need to use this new instance template for the instance group in the GKE. More information here and here .