How to increase PODS in Kubernetes

Question

I'm very new to Kubernetes, I have deployed a cluster on Kubernetes. Created a deployment and set POD's count to 2. I didn't create HPA for this deployment.

I'm using Google Cloud for this. I enabled autoscaling for the cluster. min is 2 and max is 30.

I got the OOMKilled error in my deployment.

So the question is

So only HPA can increase/decrease PODS count am I right ??. In that case, HPA based on memory and CPU is a must and should for every deployment.

Please correct me if I'm wrong.

Answer 1

You can use kubectl to change the number of pods running:

kubectl scale deployment <DEPLOYMENT NAME> --replicas=<#PODS>
kubectl scale deployment student-app --replicas=2

You can find more info at the docs page

Answer 2

Not really, OOM can be a complex issue inside your application that has nothing to do with scaling. Imagine you have a mem leak in your long running application. If the only thing you do is scaling to more pods, you'll end up just consuming more and more resources, and still running into OOM Kills on your pods.

The real solution is to have a predictable memory consumption patterns inside your applications.

Imagine an app that can use up to 64 Mb ram, but in some very rare case it can jump up to 256... well, then you need to give it 256. What if this app is running in multi-process/thread environment running N workers (ie. php-fpm) now it max mem usage is Nx256 in peaks (this is why it makes sense in many cases not to run pools of many processes in container, but scale them with HPA instead).

In the end, your memory requests/limits for a POD need to be what the application inside needs, so profiling an app, and designing it for predictability is the solution here.

And finaly, yes, if you need to scale a deployment (up/down) just use kubectl scale

Answer 3

You are correct that HPA's are pod specific. I would not say that it is a requirement for every deployment. For instance MySQL just happens to be really good at hanging on to memory, and typically applications that are written well might spike to 100% cpu while performing worker jobs. I think it really just depends on the application. L7 load balancers for instance, scale on cpu or drop packets.

Cluster scaling, however, is based on the resources block of the pod spec, specifically resource requests. Kubernetes monitors the amount of resource requests each pod on a node makes to determine how full that node is. If all nodes are full and there is a pod that is pending scheduling a new node is created.

Application developers need to not allocate memory that they do not have. If you are getting OOM killed the application is written in a way to assume that it will be able to alloc. When you write an application running inside of a docker container I believe these limits are visible to you and set by cgroups. Resource Limits can kill your container once it reaches some specified limits as well, I'm just not sure that this would display as OOM killed.

In my experience Resource Requests are something that should be provided in almost every deployment so that cluster scaling works correctly.