Opscenter LCM cluster dse not starting

Question

I have configured a Cassandra cluster through Opscenter LCM . When opscenter LCM tries to start DSE at final stage, it fails with below error.

    WARN  [main] 2018-07-20 12:24:27,504  FileSystemUtil.java:40 - JNA failed to register native C library: /tmp/jna-1073564104/jna8515864028978163146.tmp: /tmp/jna-1073564104/jna8515864028978163146.tmp: failed to map segment from shared object: Operation not permitted
{
WARN  [main] 2018-07-20 12:24:27,576  NativeLibraryLinux.java:59 - JNA not found. Native methods will be disabled.
Exception (java.lang.UnsatisfiedLinkError) encountered during startup: org.apache.cassandra.utils.NativeLibraryLinux.getpid()J
java.lang.UnsatisfiedLinkError: org.apache.cassandra.utils.NativeLibraryLinux.getpid()J
        at org.apache.cassandra.utils.NativeLibraryLinux.getpid(Native Method)
        at org.apache.cassandra.utils.NativeLibraryLinux.callGetpid(NativeLibraryLinux.java:124)
        at org.apache.cassandra.utils.NativeLibrary.getProcessID(NativeLibrary.java:429)
        at org.apache.cassandra.utils.UUIDGen.hash(UUIDGen.java:386)
        at org.apache.cassandra.utils.UUIDGen.makeNode(UUIDGen.java:367)
        at org.apache.cassandra.utils.UUIDGen.makeClockSeqAndNode(UUIDGen.java:300)
        at org.apache.cassandra.utils.UUIDGen.<clinit>(UUIDGen.java:41)
        at org.apache.cassandra.utils.ByteBufferUtil.bytes(ByteBufferUtil.java:659)
        at org.apache.cassandra.schema.TableId.toHexString(TableId.java:79)
        at org.apache.cassandra.db.Directories.<init>(Directories.java:201)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:634)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:626)
        at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:514)
        at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:372)
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:149)
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:124)
        at org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:225)
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:1554)
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1424)
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1413)
        at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:735)
        at org.apache.cassandra.cql3.QueryProcessor.executeOnceInternal(QueryProcessor.java:488)
        at com.datastax.bdp.server.DseDaemon.getLastKnownDseVersion(DseDaemon.java:692)
        at com.datastax.bdp.server.DseDaemon.checkIfRequiredUpgradeIsSkippedAndSetDseVersion(DseDaemon.java:680)
        at com.datastax.bdp.server.DseDaemon.preSetup(DseDaemon.java:654)
        at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:495)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:669)
        at com.datastax.bdp.DseModule.main(DseModule.java:91)
ERROR [main] 2018-07-20 12:24:27,581  CassandraDaemon.java:820 - Exception encountered during startup
java.lang.UnsatisfiedLinkError: org.apache.cassandra.utils.NativeLibraryLinux.getpid()J
        at org.apache.cassandra.utils.NativeLibraryLinux.getpid(Native Method) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.utils.NativeLibraryLinux.callGetpid(NativeLibraryLinux.java:124) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.utils.NativeLibrary.getProcessID(NativeLibrary.java:429) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.utils.UUIDGen.hash(UUIDGen.java:386) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.utils.UUIDGen.makeNode(UUIDGen.java:367) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.utils.UUIDGen.makeClockSeqAndNode(UUIDGen.java:300) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.utils.UUIDGen.<clinit>(UUIDGen.java:41) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.utils.ByteBufferUtil.bytes(ByteBufferUtil.java:659) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.schema.TableId.toHexString(TableId.java:79) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.Directories.<init>(Directories.java:201) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:634) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:626) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:514) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:372) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:149) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:124) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:225) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:1554) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1424) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1413) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:735) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at org.apache.cassandra.cql3.QueryProcessor.executeOnceInternal(QueryProcessor.java:488) ~[dse-db-all-4.0.0.2284.jar:4.0.0.2284]
        at com.datastax.bdp.server.DseDaemon.getLastKnownDseVersion(DseDaemon.java:692) ~[dse-core-6.0.0.jar:6.0.0]
}

The weird thing is if you manually append the cassandra-env.sh on server and put the /tmp path like- JVM_OPTS="$JVM_OPTS -Djna.tmpdir=/path_to_directory" and start DSE manually on server, it starts fine.

But what is going wrong with LCM ? I added additional jvm option for new /tmp path.

How to fix this in opscenter LCM ?

note- its DSE 6.0

Thanks

Answer 1

LCM dev here. There is something unusual about your environment, and you don't state what temp directory you configure in your manual startup sequence.

I think DSE by default might not support tmp being mounted noexec? Is there a difference in the mount options between your manual tmpdir and /tmp? Is there a difference in permissions?

Altneratively, LCM's jvm.options page as an "additional-jvm-options" list that you can add arbitrary -D options to. If you can't figure out what environmenal quirk is causing this problem, you can tell LCM to configure DSE with the extra option you've been testing manually.