Java Out of Memory Error（本机内存），进程大小限制被命中（32位linux）

Question

I am testing the performence of a web application, and have got "Out of Memory Error"(native memory).

I have test several times, and every time it "failed to allocate 83886088 bytes for Chunk::new" and died.

I print the memory size every minute, and found that the VmSize was 2924700 kB before the process died.

I think that the process size limit was hit. I change -Xmx2000m to -Xmx1900m and it's OK now.

Some questions:

1.How to comfirm that the process size limit was hit. It's not exactly 3G memory.

2.Why JVM allocate 83886088 bytes memory every time? From the exception stack, it seems that it's related to GC things.

3.How many memory should be reserved for JVM besides heap memory(-Xmx) and non-heap memory(-XX:MaxPermSize)? And how to know how many it's using now?

Thanks and sorry for my poor English.

Linux 2.6.16.60-0.83.2-bigsmp JRE 6.0_25-b06 Tomcat 7.0.37

jvm options: -Xms2000m -Xmx2000m -XX:PermSize=256M -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

/proc/meminfo:
MemTotal:     24935548 kB
MemFree:      13564968 kB

# Native memory allocation (malloc) failed to allocate 83886088 bytes for Chunk::new
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
...
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.cpp:317), pid=18217, tid=275671968

---------------  T H R E A D  ---------------

Current thread (0x105a1c00):  VMThread [stack: 0x10666000,0x106e7000] [id=18243]

Stack: [0x10666000,0x106e7000],  sp=0x106e5ae0,  free space=510k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x724710]  VMError::report_and_die()+0x2b0
V  [libjvm.so+0x2f68ef]  report_vm_out_of_memory(char const*, int, unsigned, char const*)+0x4f
V  [libjvm.so+0x1576fc]  Chunk::operator new(unsigned, unsigned)+0x5c
V  [libjvm.so+0x157c26]  Arena::grow(unsigned)+0x26
V  [libjvm.so+0x6457e9]  resource_allocate_bytes(unsigned)+0x49
V  [libjvm.so+0x3a03ef]  GenericGrowableArray::raw_allocate(int)+0xbf
V  [libjvm.so+0x3402ee]  GrowableArray<oopDesc*>::grow(int)+0x3e
V  [libjvm.so+0x3a62b1]  FindInstanceClosure::do_object(oopDesc*)+0x51
V  [libjvm.so+0x2ae571]  CompactibleFreeListSpace::object_iterate(ObjectClosure*)+0x51
V  [libjvm.so+0x38d066]  GenerationObjIterateClosure::do_space(Space*)+0x16
V  [libjvm.so+0x2d6498]  ConcurrentMarkSweepGeneration::space_iterate(SpaceClosure*, bool)+0x18
V  [libjvm.so+0x38be22]  Generation::object_iterate(ObjectClosure*)+0x22
V  [libjvm.so+0x2d9f9d]  ConcurrentMarkSweepGeneration::object_iterate(ObjectClosure*)+0x4d
V  [libjvm.so+0x37fd65]  GenCollectedHeap::object_iterate(ObjectClosure*)+0x55
V  [libjvm.so+0x3a61fe]  HeapInspection::find_instances_at_safepoint(klassOopDesc*, GrowableArray<oopDesc*>*)+0x3e
V  [libjvm.so+0x6eca91]  ConcurrentLocksDump::dump_at_safepoint()+0xf1
V  [libjvm.so+0x6e7561]  Threads::print_on(outputStream*, bool, bool, bool)+0x201
V  [libjvm.so+0x734927]  VM_PrintThreads::doit()+0x27
V  [libjvm.so+0x734576]  VM_Operation::evaluate()+0x46
V  [libjvm.so+0x733a23]  VMThread::evaluate_operation(VM_Operation*)+0x83
V  [libjvm.so+0x733c90]  VMThread::loop()+0x190
V  [libjvm.so+0x733780]  VMThread::run()+0x80
V  [libjvm.so+0x5e294e]  java_start(Thread*)+0x14e
C  [libpthread.so.0+0x54ab]  short+0x8b

VM_Operation (0x0caff000): PrintThreads, mode: safepoint, requested by thread 0x09ec3800


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
...//209 Java Threads

Heap
 par new generation   total 147456K, used 126550K [0x161f0000, 0x201f0000, 0x201f0000)
  eden space 131072K,  92% used [0x161f0000, 0x1d869d18, 0x1e1f0000)
  from space 16384K,  31% used [0x1e1f0000, 0x1e70be78, 0x1f1f0000)
  to   space 16384K,   0% used [0x1f1f0000, 0x1f1f0000, 0x201f0000)
 concurrent mark-sweep generation total 1884160K, used 1697819K [0x201f0000, 0x931f0000, 0x931f0000)
 concurrent-mark-sweep perm gen total 262144K, used 61769K [0x931f0000, 0xa31f0000, 0xb31f0000)

Code Cache  [0xb391f000, 0xb44b7000, 0xb691f000)
 total_blobs=3924 nmethods=3719 adapters=157 free_code_cache=38272704 largest_free_block=9600

Answer 1

print the memory size every minute, and found that the VmSize was 2924700 kB before the process died.

AFAIK On Linux the OS uses about 1 GB. You need virtual memory for thread stacks, shared libraries, Perm Gen, memory mapped files, and any native resources.

If you are getting anywhere near this limit (or if you just have a 64-bit processor) you really should be using a 64-bit OS and JVM. I would use Java 6 update 45, which is end of free support, if not Java 7 update 25.

Answer 2

process size limit

Modern Linux OSs can use more than 4GB of RAM when in the 32bit version by using something called Physical Address Extension (PAE) .

But if you don't do anything, 3GB seems like a reasonable limit that processes may hit.

That said, this is completely unrelated to your case. The error is not from the kernel (you didn't get a core dump) but from the Java runtime.

The reason why it happens in the GC is that some other thread asked for another 83886088 bytes of free memory and there wasn't any left. So the VM started the GC to make room. Unfortunately, all objects were still in use -> out of memory error.

Why JVM allocate 83886088 bytes memory every time?

Software should be deterministic, that is it should produce the same results every time you run it.

Look at the other threads. One of them just tried to allocate exactly this memory.

How many memory should be reserved for JVM besides heap memory(-Xmx) and non-heap memory(-XX:MaxPermSize)?

That depends on what your application does. But usually, out of memory means you have a memory leak (ie something holds on to big objects that it doesn't really need anymore).

Use a profiler like MAT or YourKit to determine what that could be.

And how to know how many it's using now?

Use jconsole or VisualVM to monitor the memory usage.

Related:

• http://memoryanalyzer.blogspot.co.uk/ , especially http://memoryanalyzer.blogspot.co.uk/2008/05/automated-heap-dump-analysis-finding.html
• http://www.oracle.com/technetwork/java/javase/memleaks-137499.html
• Oracle docs: Troubleshooting Memory Leaks

Answer 3

Try to change the stack size for a thread, use the -Xss=N flag (eg, - Xss=256k ). By default for linux the size is 320 KB (In 64 bit machine the size is 1 MB). Generally, Out of Native Memory happens because of following reasons:

Please note that changing the stack size will help in case of 1 and 2.

1. In a 32-bit JVM, the process is at its 4 GB (or less, depending on the OS) maximum size.
2. The system has actually run out of virtual memory.
3. On Unix-style systems, the user has already created (between all programs she is running) the maximum number of processes configured for her login. Individual threads are considered a process in that regard.

Hope this helps.

Regards, Pinaki