Maximum time a docker containers can be used without respinning?

Question

I have this jenkins master-worker docker slave configuration that runs java test cases.

Once the vapp is up with master and 3 slave configuration, the vapp is used for more than 2 months to run the test cases.

At a time nearly 500 testcases are ran which takes 1 hr to complete the execution and there are n number of vapps like this and depending on availability vapp is picked to execute the test cases.

After using this vapp for more than a month, I see the linux kernel is actually killing the master container due to Out of Memory and this is not common to all the vapps but only in a few vapps.

All the cache and memory managements are done and eventhough the containers are getting killed.

My question is like is it recommended to use the vapp for more than a month continuously?

I asked to respin the vapps which means respin the containers after a month.

What is the best possible solution to this?

Answer 1

It seems some of your containerized applications suffer from memory leaks. Operationally your behaviour looks good already: The container gets killed and restarted if it hits configured memory limits. You could also limit the lifetime of the container as you are questioning anyway, and the real answer is in the amount of resources you are willing to spend. Would it hurt to spin up a new container for every one hour test?

If this is not good enough, you'd probably have to investigate with the developer of the application to find and remove the memory leak.