"Insufficient memory" on Jenkins server startup

Question

First time user of Jenkins here, and having a bit of trouble getting it started. From the Linux shell I run a command like:

java -Xms512m -Xmx512m -jar jenkins.war

and consistently get an error like:

# There is insufficient memory for the Java Runtime Environment to continue.
# pthread_getattr_np
# An error report file with more information is saved as:
# /home/twilliams/.jenkins/hs_err_pid36290.log

First, the basics:

• Jenkins 1.631
• Running via the jetty embedded in the war file
• OpenJDK 1.7.0_51
• Oracle Linux (3.8.13-55.1.5.el6uek.x86_64)
• 386 GB ram
• 40 cores

I get the same problem with a number of other configurations as well: using Java Hotspot 1.8.0_60, running through Apache Tomcat, and using all sorts of different values for -Xms/-Xmx/-Xss and similar options.

I've done a fair bit of research and think I know what the problem is, but am at a loss as how to solve it. I suspect that I'm running into the virtual memory overcommit issue mentioned here ; the relevant bits from ulimit:

--($:)-- ulimit -a
...
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) 8388608
stack size              (kbytes, -s) 8192
virtual memory          (kbytes, -v) 8388608
...

If I double the virtual memory limit as root, I can start Jenkins, but I'd rather not run Jenkins as the root user.

Another workaround: a soon-to-be-decommissioned machine with 48 GB ram and 24 cores can start Jenkins without issue, though (I suspect) just barely: according to htop, its virtual memory footprint is just over 8 GB. I suspect, as a result, that the memory overcommit issue is scaling with the number of processors on the machine, presumably the result of Jenkins starting a number of threads proportional to the number of processors present on the host machine. I roughly captured the thread count via ps -eLf | grep jenkins | wc -lps -eLf | grep jenkins | wc -l ps -eLf | grep jenkins | wc -l and found that the thread count spikes at around 114 on the 40 core machine, and 84 on the 24 core machine.

Does this explanation seem sound? Provided it does...

1. Is there any way to configure Jenkins to reduce the number of threads it spawns at startup? I tried the arguments discussed here but, as advertised, they didn't seem to have any effect.
2. Are there any VMs available that don't suffer from the overcommit issue, or some other configuration option to address it?

The sanest option at this point may be to just run Jenkins in a virtualized environment to limit the resources at its disposal to something reasonable, but at this point I'm interested in this problem on an intellectual level and want to know how to get this recalcitrant configuration to behave.

Edit

Here's a snippet from the hs_error.log file, which guided my initial investigation:

# There is insufficient memory for the Java Runtime Environment to continue.
# pthread_getattr_np
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.

Here are a couple of command lines I tried, all with the same result.

• Starting with a pitiful amount of heap space:
  • java -Xms2m -Xmx2m -Xss228k -jar jenkins.war
• Starting with significantly more heap space:
  • java -Xms2048m -Xmx2048m -Xss1m -XX:ReservedCodeCacheSize=512m -jar jenkins.war
• Room to grow:
  • java -Xms2m -Xmx1024m -Xss228k -jar jenkins.war

A number of other configurations were tried as well. Ultimately I don't think that the problem is heap exhaustion here - it's that the JVM is trying to reserve too much virtual memory for itself (in which to store the heap, thread stacks, etc) than allowed by the ulimit settings. Presumably this is the result of the overcommit issue linked earlier, such that if Jenkins is spawning 120 threads, it's erroneously trying to reserve 120x as much VM space as the master process originally occupied.

Having done what I could with the other options suggested in that log, I'm trying to figure out how to reduce the thread count in Jenkins to test the thread VM overcommit theory.

Edit #2

Per Michał Grzejszczak, this is an issue with the glibc distributed with Red Hat Enterprise Linux 6 as discussed here . The issue can be worked around via explicit setting of the environment variable MALLOC_ARENA_MAX , in my case export MALLOC_ARENA_MAX=2 . Without explicit setting of this variable, the JVM will apparently attempt to spawn (8 x cpu core count) threads, each consuming 64M. My 40 core case would have required northward of 10 gigs of virtual ram, exceeding (by itself) the ulimit on my machine of 8 gigs. Setting this to 2 reduces VM consumption to around 128 megs.

Answer 1

Jenkins memory footprint is related more to the number and size of projects it is managing than the number of CPUs or available memory. Jenkins should run fine on 1GB of heap memory unless you have gigantic projects on it.

You may have misconfigured the JVM though. -Xmx and -Xms parameters govern heap space JVM can use. -Xmx is a limit for heap memory, -Xms is a starting value for heap memory. Heap is a single memory area for entire JVM. You can easily monitor it by various tools like JConsole or VisualVM.

On the other hand -Xss is not related to heap. It is the size of a thread stack for all threads in this JVM process. As Java programs tend to create numerous threads setting this parameter too big can prevent your program from launching. Typically this value is in the range of 512 kb . Entering here 512 m instead makes it impossible for JVM to start. Make sure your settings do not contain any such mistakes (and post your memory config too).