k8s service internal bench

Question

I'm currently trying to benchmark different services deployment in a k8s cluster.

It's all here , but I'll save you the dig in. The service itself is ultra simple, as well as the deployment (basically a http get), and I added an hpa, which work well.

The bench is running inside the same cluster, on a specific node.

The bench runs ok, everything seems to be working as expected.

If I take one of the results and give an extract here:

NAME                                   CPU(cores)   MEMORY(bytes)   
bencher-deployment-cf89ddc67-bwwgb     197m         4Mi             
go-induzo-deployment-d6cbc56c6-97j62   1m           7Mi             
go-induzo-deployment-d6cbc56c6-c2w24   0m           7Mi             
go-induzo-deployment-d6cbc56c6-jh768   0m           7Mi             
go-induzo-deployment-d6cbc56c6-mfdhb   0m           6Mi             
go-induzo-deployment-d6cbc56c6-mh6mt   820m         11Mi            
go-induzo-deployment-d6cbc56c6-pktn4   939m         11Mi            
go-induzo-deployment-d6cbc56c6-vdjjj   1m           5Mi             
go-induzo-deployment-d6cbc56c6-x64jw   893m         11Mi            
go-induzo-deployment-d6cbc56c6-zhsp7   0m           5Mi             
go-induzo-deployment-d6cbc56c6-zvf9m   0m           5Mi    

     

As you can notice, the hpa is triggered, and scaled to 10 pods.

But as you can notice as well, the load is only balanced between 3 pods, it's not using the other ones. It seems it can only use one pod per node, and not the other ones.

Anything I would have forgotten? Is it expected? Do I need to add a load balancer service to actually leverage all pods?

Answer 1

I forgot to make sure there is no Keep Alive in the benchmark tool.

in the case of wrk, which I'm using: -H "Connection: Close"