Cassandra-JVM OOM直接缓冲区错误

Question

We have a Datastax Enterprise cluster of the following configuration:

java version "1.8.0_181"
DataStax Enterprise Version: 6.0.0

Number of Nodes: 3
Node Listing:
Name: localhost - xx.xx.xx.01
Cassandra Version: 4.0.0.2284
DataStax Enterprise Version: 6.0.0
Available Memory: 15586 MB
Number of CPU Cores: 4
Operating System: linux
Space Used: 5 GB / 125 GB
Name: localhost - xx.xx.xx.02
Cassandra Version: 4.0.0.2284
DataStax Enterprise Version: 6.0.0
Available Memory: 15586 MB
Number of CPU Cores: 4
Operating System: linux
Space Used: 5 GB / 125 GB
Name: localhost - xx.xx.xx.03
Cassandra Version: 4.0.0.2284
DataStax Enterprise Version: 6.0.0
Available Memory: 15586 MB
Number of CPU Cores: 4
Operating System: linux
Space Used: 6 GB / 125 GB

Keyspace size - 1.34 GB

Yesterday, We'd a lot of OOM Errors on 1 of 3 nodes and similar errors came on the other nodes after restarting the first one. Error details:

ERROR [CompactionExecutor:4477] 2018-08-29 13:23:00,320  JVMStabilityInspector.java:117 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_181]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_181]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_181]
        at org.apache.cassandra.io.compress.BufferType$2.allocate(BufferType.java:39) ~[dse-db-all-4.0.0.2284.jar:6.0.0]
        at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:89) ~[dse-db-all-4.0.0.2284.jar:6.0.0]
        at org.apache.cassandra.io.sstable.format.trieindex.TrieIndexSSTableWriter.<init>(TrieIndexSSTableWriter.java:100) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.io.sstable.format.trieindex.TrieIndexFormat$WriterFactory.open(TrieIndexFormat.java:110) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:108) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.switchCompactionLocation(DefaultCompactionWriter.java:71) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.maybeSwitchWriter(CompactionAwareWriter.java:182) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:144) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:210) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:92) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_181]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]

This error seems related with Java.nio Direct Buffer Cache which have no upper bound limit and keeps growing until OOM incident occurs. ( https://support.datastax.com/hc/en-us/articles/360000863663-JVM-OOM-direct-buffer-errors-affected-by-unlimited-java-nio-cache ). We're seeing continuously increase in memory utilization on all of our Cassandra nodes. This behaviour is still persist even after restarting Cassandrandra nodes.

JVM Config:

-XX:+AlwaysPreTouch
-Dcassandra.disable_auth_caches_remote_configuration=false
-Dcassandra.expiration_date_overflow_policy="REJECT"
-Dcassandra.force_default_indexing_page_size=false
-Dcassandra.join_ring=true
-Dcassandra.load_ring_state=true
-Dcassandra.write_survey=false
#-XX:ConcGCThreads=
-ea
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:GCLogFileSize=10M
-XX:+HeapDumpOnOutOfMemoryError
#-Xmsauto
#-XX:InitiatingHeapOccupancyPercent=
-Dio.netty.eventLoop.maxPendingTasks=65536
-Djava.net.preferIPv4Stack=true
-XX:MaxGCPauseMillis=500
#-Xmxauto
-XX:NumberOfGCLogFiles=10
-Dsun.nio.PageAlignDirectMemory=true
#-XX:ParallelGCThreads=
-Xss256k
-XX:+PerfDisableSharedMem
-XX:+PreserveFramePointer
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintHeapAtGC
-Dcassandra.printHeapHistogramOnOutOfMemoryError=false
-XX:+PrintPromotionFailure
-XX:+PrintTenuringDistribution
-XX:+ResizeTLAB
-XX:-RestrictContended
-XX:StringTableSize=1000003
-XX:ThreadPriorityPolicy=42
-XX:+UnlockDiagnosticVMOptions
-XX:+UseGCLogFileRotation
-XX:+UseThreadPriorities
-XX:+UseTLAB





-XX:+UseG1GC


JVM_ON_OUT_OF_MEMORY_ERROR_OPT="-XX:OnOutOfMemoryError=kill -9 %p"



-Dcom.sun.management.jmxremote.authenticate=false
-Dcassandra.jmx.local.port=7199

Answer 1

Can you give more info about the heap info when the oom come? old gen, eden, s1 and so on.

1. check your JVM arguments, for example:-XX:+DisableExplicitGC, if this args is set to false, then System.gc() will do nothing.

2. Is your jvm has fgc? check your code for that where DirectByteBuffer object is allocated.